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ABSTRACT 


The  concept  of  using  simulation  for  training  is  not  new  in  the  aviation  experience. 
Flight  simulators  have  been  in  existence  since  the  early  1 920s  and  have  evolved  from 
non-mechanical,  tethered  aircraft  to  high  fidelity,  multi-million  dollar  motion  platforms. 
In  the  demanding  and  dynamic  environment  of  aviation,  errors  can  be  catastrophic.  It  is 
not  surprising  that  both  the  civilian  and  military  communities  have  made  simulation 
training  a  critical  component  of  their  aviation  training  programs. 

Despite  the  wide  variety  of  simulation  training  efforts  being  pursued  by  military 
and  civilian  aviation  alike,  there  is  limited  research  on  the  subject  of  low-cost  team 
training,  particularly  in  the  area  of  situation  awareness.  Most  research  has  been  directed 
toward  individual  task  performance.  Only  in  recent  years  has  additional  emphasis  been 
placed  on  crew  resource  management  (CRM).  Even  less  literature  exists  on  the  concept 
of  aircraft  to  aircraft  team  training  -  an  wfercockpit  training  model  as  opposed  to  an 
/wtracockpit  training  model.  Simulation  being  used  by  the  U.S.  Army  that  is  team  based 
focuses  mostly  on  commander  and  battle  staff  training. 

There  is  a  clear  lack  of  research  concerning  the  suitability  of  using  low-cost 
simulation  for  training  U.S.  Army  aviators  in  team  collective  tasks  and  situational 
awareness  (SA).  The  purpose  of  this  research  is  to  develop  an  assessment  tool  and 
conceptual  model,  tailored  to  emulate  Army  aviation  platforms,  that  can  be  used  in 
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assessing  the  suitability  of  this  approach  for  achieving  Army  aviation  team  collective  task 
and  situational  awareness  training  goals.  Lessons  learned  and  areas  for  future  research 
are  also  discussed. 
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“In  1901,  my  father  had  a  dream  of  building  an  aircraft  that  could  take-off  and  land 
vertically.  In  193 1,  he  patented  a  design  of  a  now  familiar  helicopter  layout  -  a  single 
large  main  rotor  and  a  small  anti-torque  tail  rotor.  In  the  summer  of  1939,  he  began  in 
earnest  to  develop  the  world’s  first  successful,  practical  helicopter. 

This  steel  tube  and  open  cockpit  vehicle,  that  he  called  the  VS  300,  flew  under  tethered 
flight  with  my  father  at  the  controls  for  only  a  few  minutes.  Within  a  year,  the 
experimental  helicopter  would  remain  airborne  for  longer  periods  of  time. 

When  asked  how  he  succeeded  in  making  a  helicopter  work  when  so  many  others  had 
failed,  he  said,  ‘I  do  not  know.  Maybe  it  was  just  because  nobody  told  me  that  it  couldn’t 
be  done!”’ 


-  Igor  Sikorsky,  Jr. 
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CHAPTER  1 


INTRODUCTION 

1.1  Background 

A  common  credo  used  in  the  U.S.  Army  professes  “as  you  train,  so  shall  you 
fight”.  Training  the  force  to  win  on  the  battlefield  is  critical  to  ensuring  the  coordinated 
and  accurate  execution  of  doctrine  in  the  fluid  and  ever-changing  environment  of  modem 
warfare.  Lack  of  training  results  in  errors  that  will,  undoubtedly,  have  consequences  on 
the  battlefield.  In  the  dynamic  environment  of  aviation,  these  consequences  can  have 
disastrous  and  even  fatal  results.  For  this  reason,  it  is  not  surprising  that  both  the  civilian 
and  military  communities  have  incorporated,  and  relied  heavily  upon,  simulation  as  a 
critical  component  of  their  aviation  training  programs.  Advancements  in  technology 
have  greatly  enhanced  this  ability.  Today  it  is  possible  to  replicate  virtually  any 
environment  or  element  of  equipment  for  use  in  training  simulation.  Still,  as  training 
budgets  continue  to  become  smaller  and  smaller,  the  need  to  find  low-cost  alternatives  for 
training  has  become  imperative. 

Most  training  research  efforts  in  aviation,  particularly  from  World  War  II  to  the 
present,  have  focused  on  the  use  of  simulation  to  enhance  pilot  training.  The  concept  of 
using  simulation  for  training  is  not  new  in  the  aviation  experience.  Flight  simulators 
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have  evolved  from  non-mechanical,  tethered  aircraft  of  the  1920s,  to  high  fidelity,  multi¬ 
million  dollar  motion  platforms  with  six  degrees  of  freedom  ( dj ).  A  majority  of  this 
research  has  been  directed  toward  individual  task  performance  and  psychomotor  skills. 
However,  in  light  of  the  many  military  and  commercial  airline  accidents  that  have 
occurred  due  to  crew  error,  the  emphasis  in  recent  years  has  shifted  towards  team  training 
in  the  cockpit,  otherwise  known  as  crew  resource  management  (CRM).  Although  there  is 
much  literature  concerning  the  subjects  of  aircrew  training,  team  processes,  and  team 
training  in  general,  little  research  has  been  done  to  extend  these  concepts  for  aircraft  to 
aircraft  applications,  thus  developing  an  /wtracockpit  training  model  as  opposed  to  an 
/wfercockpit  training  model. 

Even  less  literature  exists  concerning  the  suitability  of  using  low  cost  simulation 
to  train  team  processes,  such  as  situational  awareness  (SA),  and  behaviors  between 
aircraft,  specifically  in  U.S.  Army  aviation  (personal  communication  with  Mr.  Delashaw, 
May  28,  1999).  There  have  been  aviation-related  studies  conducted  in  other  services  that 
explore  comparisons  between  individual  and  team  knowledge  structures  and  team 
performance  in  multi-service  distributed  training  exercises  (Gualtieri,  Fowlkes,  &  Ricci, 
1996;  Oser,  Dwyer,  &  Fowlkes,  1995).  Although  the  cognitive  processes  studied  were 
very  applicable  in  terms  of  aviation  team  training,  they  were  conducted  using  medium  to 
high  fidelity,  geographically  remote  simulators  and  focused  more  on  crew  behaviors 
rather  than  the  value  of  intercockpit  training. 
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1.2  Current  Army  Aviation  Training  Practices 


To  ensure  that  army  aviators  maintain  proficiency  in  flying  skills  and  applied 
operations,  the  army  requires  that  pilots  pass  an  annual  proficiency  and  readiness  test 
(APART).  This  test  consists  of  a  series  of  written  and  flight  evaluations,  in  which  the 
aviator  must  demonstrate  an  acceptable  level  of  aircraft  knowledge,  flying  skills,  and 
aviation  related  subjects  (DA,  1992).  The  instrument  flight  evaluation  can  be  performed 
in  a  flight  simulator  if  available.  The  tasks  that  each  aviator  must  perform  are  outlined  in 
great  detail,  with  their  associated  descriptions  of  task,  condition,  and  standards,  in  each 
aircraft’s  specific  Aircrew  Training  Manual  (ATM)  (DA,  in  press).  Pilots  are  also 
required  to  complete  a  minimum  number  of  iterations  of  certain  tasks  in  a  semi-annual 
and  annual  period.  Despite  this  emphasis,  there  is  very  little  focus  or  specificity  given  to 
team  or  inter  cockpit  training. 

Chapter  6  of  the  ATM  does  specifically  address  crew  coordination,  but  the  subject 
matter  primarily  focuses  on  those  elements  that  make  an  effective  crew  within  the  cockpit 
as  opposed  to  between  aircraft.  Crews  frequently  use  simulators  to  reinforce  CRM 
concepts,  but  these  scenarios  represent  only  single  ship  operations.  This  lack  of  emphasis 
is  evident  in  the  non-specific,  and  often  vague,  references  to  any  task  involving  multiple 
aircraft  operations.  In  the  UH-60  Blackhawk  helicopter  ATM,  for  example,  there  is  only 
one  task,  labeled  “Perform  multi-aircraft  operations”  (in  press),  that  addresses  multiple 
aircraft  interaction  in  terms  of  a  required  annual  task.  Due  to  the  complex  nature  of 
determining  how  a  flight  of  aircraft  and  the  individual  pilots  within  it  have  “met  the 
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standard”,  the  task  guidance  basically  bequeaths  the  conduct  of  the  task  and  what  is  to  be 
performed  to  the  discretion  of  the  instructor  pilot(s).  The  evaluation  of  this  task  requires 
the  use  of  two  aircraft  or  more.  Over  time,  this  can  prove  to  be  a  costly  endeavor.  PC- 
based  simulation  is  seldom  used  as  a  rehearsal  tool,  however,  aviators  do  use  PCs  for 
various  functions  of  mission  planning,  e.g.  weather,  performance  planning,  etc. 
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1.3  Simulation  and  PC-Based  Training 


PC-based  simulation  already  exists  in  many  forms  in  the  gaming  community  and 
has  been  used  to  investigate  team  processes  (Stout,  Salas,  &  Carson,  1994).  Additionally, 
they  have  been  integrated  into  the  initial  phases  of  general  aviation  in  the  form  of 
interactive  courseware  and  instrument  training  devices.  However,  bringing  PC-based 
training  simulation  into  aviation  team  training  alone  will  not  define  its  applicable  benefits 
for  aviation  team  training  use. 

As  indicated  in  previous  discussions  concerning  annual  aircrew  training,  U.S. 
Army  aviation  currently  has  little  in  the  way  of  PC-based  aviation  team  training.  Though 
simulation  is  used  extensively  in  army  aviation  training,  these  efforts  focus  primarily  on 
aircraft  qualification  training,  CRM,  battle  staff  training,  and  the  linkage  of  high  cost 
systems  to  other  army  simulation  centers  for  the  purpose  of  conducting  combined  arms 
exercises.  There  has  been  minimal  emphasis  placed  on  using  PC-based  simulation  to  link 
aircraft  together  for  training  team  SA,  and  team  processes  in  multi-ship  helicopter 
operations  (personal  communication  with  Mr.  Delashaw,  May  28,  1999). 

There  is  no  doubt  that  simulation  is  prevalent  in  almost  every  facet  of  the  Army  - 
from  simulation  based  acquisition  to  training,  and  it  is  most  certainly  at  the  forefront  of 
army  aviation  training  (DA,  in  press).  Still,  focus  on  PC-based  simulation  training  is 
relatively  limited,  and  what  simulation  that  is  available  is  extremely  high  fidelity  and 
definitely  high  cost  (Fort  Rucker,  1999).  Training  the  force  is,  unquestionably,  an 
integral  part  of  the  Army’s  readiness,  yet  this  priority  is  primarily  served  by  very  large- 
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scale  distributed  training  simulation  exercises  connecting  geographically  separated 
military  facilities. 
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1 .4  Current  Army  Aviation  Simulation  Efforts 


Most  efforts  to  upgrade  aviation  training  and  its  interaction  with  other  Army 
battlefield  operating  systems  occur  at  the  U.S.  Army  Aviation  Center,  located  at  Fort 
Rucker,  Alabama.  One  individual  who  has  been  close  to  these  developments  is  Mr.  Jim 
Delashaw,  a  representative  of  the  Program  Executive  Office  for  Aviation  (PEO  Avn)  who 
interfaces  with  the  Directorate  for  Training,  Doctrine,  and  Simulation  (DOTDS)  at  Fort 
Rucker.  According  to  Mr.  Delashaw,  there  is  an  ongoing  project  at  the  Aviation  Test  Bed 
(AVTB)  that  involves  configuring  PC-based  systems  to  run  synthetic  battlefield  entities 
and  terrain  and  also  run  image  generators  that  can  show  these  entities  to  the  crew.  These 
type  of  systems  are  referred  to  as  Fully  Reconfigurable  Engineering  Devices  (FREDs), 
which  were  originally  developed  for  the  Simulation  Net  (SIMNET)  program,  and  still 
represent  the  primary  training  system  used  for  team  and  unit  training  (personal 
communication  with  Mr.  Delashaw,  May  28,  1999). 

Another  team  training  simulation  system  currently  in  development  is  the  Aviation 
Combined  Arms  Tactical  Trainer  (AVCATT).  When  fully  developed,  this  particular 
training  device  will  link  with  other  elements  of  a  larger  Combined  Arms  Tactical  Trainer 
(CATT)  community  that  includes  similar  engineer,  air  defense,  and  fire  support  trainers 
(STRICOM,  1999).  This  system,  however,  is  definitely  high  cost,  and  at  least  moderate 
in  its  fidelity. 

The  only  low-cost  simulation  training  device  that  has  been  developed  for  Army 
aviation  team  training  is  the  Aviation  Reconfigurable  Manned  Simulator  (ARMS) 
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(STRICOM,  1 999).  It  is  intended  to  be  fielded  to  the  Army  National  Guard  ( ARNG)  in 
December  1999  and  does  focus  on  training  aviation  collective  tasks.  However,  it  is  not 
PC-based,  and  there  is  little  evidence  at  this  time  to  suggest  whether  aircraft-to-aircraft 
team  training  is  emphasized. 

Though  the  use  of  simulation  for  aviation  training  exists  in  many  forms  in  the 
military  community,  the  literature  suggest  that  the  development  of  low-cost  simulation 
for  Army  aviation  team  training  is  limited.  However,  simply  introducing  a  platform,  such 
as  a  PC-based  system,  into  the  picture  is  not  sufficient  for  validating  its  training 
effectiveness.  An  understanding  of  how  teams  share  knowledge  and  how  tasks  are 
analyzed  is  also  critical  to  ensuring  the  development  of  a  successful  training  device.  This 
realization  begs  several  questions:  1)  Can  current  models  for  team  training  be  extended 
for  intercockpit  team  training  or  is  the  formulation  of  a  new  model  required?,  2)  What 
tasks  and  behaviors  are  necessary  to  achieve  team  SA  in  Army  helicopter  pilots?,  and  3) 
Is  low-cost,  PC-based  simulation  a  valid  tool  for  training  these  behaviors? 
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CHAPTER  2 


TEAM  TRAINING,  SITUATION  AWARENESS,  AND  FIDELITY 

2.1  Teams  and  Teamwork  Defined 


Before  launching  into  a  detailed  discussion  of  team  training  and  its  functions  in 
aviation  training,  it  is  necessary  to  have  a  clear  understanding  of  what  a  team  is.  In  a 
variety  of  sources,  the  definition  of  team  training  has  taken  on  many  forms.  However, 
the  definition  put  forth  by  Salas,  Dickinson,  Converse,  and  Tannenbaum  (1992)  is  one 
that  is  generally  accepted,  and  can  be  best  applied,  in  the  operational  and  academic 
communities.  They  define  a  team  as  “a  distinguishable  set  of  two  or  more  people  who 
interact  dynamically,  interdependently,  and  adaptively  toward  a  common  and  valued 
goal/objective/mission,  who  have  limited  life  span  and  membership”  (p.  4).  It  is 
important  to  note  that  teams  are  distinguishable  from  groups  in  that  groups  may  be 
required  to  interact,  but  do  not  necessarily  need  to  coordinate  actions  for  mission 
accomplishment  (Salas  &  Cannon-Bowers,  in  press). 

A  tactical  decision-making  team  such  as  a  tank  platoon  or  a  flight  of  helicopters 
often  has  other  characteristics  that  can  have  a  dramatic  effect  on  the  level  of  SA  and, 
ultimately  performance,  that  the  team  can  achieve.  The  operational  tactical  teams  of 
today  follow  a  hierarchical  structure  and  must  constantly  adjust  to  rapid  changes  in  the 
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environment  (Orasanu  &  Salas,  1993).  In  the  case  of  military  helicopter  operations,  this 
normally  involves  responding  to  extremely  high  cockpit  workloads,  adverse  weather 
conditions,  time  constraints,  mission  changes,  and  the  threat  of  enemy  engagement. 

In  the  field  of  aviation,  we  tend  to  associate  team  training  with  crew  coordination 
or  CRM  (see  Baker  &  Salas,  1992;  Orasanu  &  Salas,  1993;  Prince  &  Salas,  1999;  Stout, 
Salas,  &  Carson,  1994).  However,  these  ideas  can  be  extended  to  include  multi-ship 
operations  in  which  multiple  crews  operate  in  separate  aircraft,  but  towards  the  same 
objectives  -  mission  accomplishment  and  personnel  safety.  This  type  of  training  would 
focus  primarily  on  an  zwfercockpit  training  environment/model  as  opposed  to  an 
intracockpit  one.  In  this  case  the  team  behaviors,  such  as  communication  and  decision¬ 
making,  can  be  projected  beyond  the  cockpit  to  incorporate  factors  associated  with  not 
being  able  to  see  other  members  of  the  team  due  to  aircraft  separation. 

From  our  definition  of  a  team,  we  can  derive  a  similar  definition  of  teamwork. 
Stout  et  al.  (1996)  summarize  teamwork  “as  an  identifiable  set  of  behaviors,  cognitions, 
and  attitudes  that  contribute  to  the  team’s  overall  functioning”  (p.  89).  Although  a  great 
deal  of  empirical  studies  have  focused  on  teamwork  and  team  behaviors,  there  has  not 
been  much  attention  given  to  how  the  components  of  teamwork  and  their  corresponding 
interactive  processes  affect  decision-making  and  SA  (1996).  The  proper  implementation 
of  shared  mental  models  and  cognitive  task  analysis  play  important  roles  and  may 
contribute  to  the  training  and  evaluation  of  these  components. 
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2.2  Shared  Mental  Models 


Key  to  maximizing  the  training  and  evaluation  of  team  behaviors  and  processes  is 
an  understanding  of  how  mental  models  contribute  to  the  overall  learning  process.  By 
combining  the  descriptions  of  various  experts  (Orasanu,  1990;  Cannon-Bowers,  Salas,  & 
Converse,  1993)  we  can  define  a  mental  model  as  a  psychological  or  mental 
representation  of  specific  components  of  knowledge  that  exist  within  a  particular 
environment.  A  mental  model  may  also  represent  the  relationships  between  various 
elements  of  knowledge  (Klemoski  &  Mohammed,  1994).  Likewise,  “shared”  mental 
models  are  mental  models  that  team  members  hold  in  common.  For  our  purposes  we  will 
focus  on  the  use  of  shared  mental  models  in  developing  team  SA.  The  concept  of  team 
SA  will  be  discussed  in  greater  detail  in  a  later  section  concerning  SA. 

In  an  effort  to  fill  the  gap  in  team  training,  shared  mental  models  have  been 
brought  to  the  forefront  of  team  training  studies.  It  is  clear  that  these  models  are 
necessary  because  they  enable  team  members  to  effectively  handle  situational  change  and 
predict  needs  and  actions  of  other  team  members  (Stout,  et  al.,  1996).  To  assist  in 
determining  how  shared  mental  models  contribute  to  the  team  training  process,  we  must 
ask  ourselves  what  type  of  knowledge  team  members  need  to  share  in  order  to  complete 
their  tasks  (1996).  Team  member  goals,  attitudes,  roles,  and  tasks  all  contribute  towards 
effective  task  accomplishment  (Canon-Bowers,  Salas,  &  Converse,  1993).  Converse  and 
Kahler  (1992)  have  suggested  that  there  are  three  forms  of  knowledge  that  team  members 
can  hold  in  common:  declarative,  procedural,  and  strategic. 
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Declarative,  or  factual,  knowledge  concerns  the  rules  and  facts  that  govern  a 
situation,  to  include  equipment,  system  components,  team  members,  and  their 
relationships  (Stout,  Cannon-Bowers,  &  Salas,  1994).  Procedural  knowledge  is  more  of  a 
sequential  form  of  knowledge  in  which  there  is  an  understanding  of  task  relationships  and 
how  the  actions  of  other  team  members  affect  these  relationships  (1994).  Stout,  et  al. 
(1996)  point  out  that  it  is  important  that  certain  portions  of  declarative  and  procedural 
knowledge  must  be  shared  among  team  members: 

. . .  [I]t  is  probably  not  necessary  for  team  members  to  share  an 
understanding  of  detailed  information  regarding  each  other’s  equipment. 

However,  one  could  speculate  that  team  members  should  understand:  1) 
what  important  information  a  fellow  team  member  needs  from  them  in 
order  for  him/her  to  accomplish  his/her  tasks  and  2)  when  in  a  task 
sequence  this  information  should  be  presented,  (p.  90) 

In  an  aviation  context,  we  can  demonstrate  this  point  with  an  example  of  a  tandem 
cockpit  configuration,  such  as  that  of  an  AH-64  Apache  helicopter,  in  which  the  pilot  and 
the  copilot/gunner  cannot  see  each  other’s  displays  or  cockpit  environment.  Neither 
needs  to  know  every  detail  of  what  is  going  on  in  each  other’s  environment,  but  each 
needs  to  have  knowledge  of,  and  be  aware  of,  each  other’s  functions  so  as  to  assist  or 
“cover”  each  other  as  the  tactical  situation  evolves. 

Strategic  knowledge,  according  to  Converse  and  Kahler  (1992),  serves  as  the 
basis  for  problem  solving.  It  includes  such  information  as  “knowledge  of  the  context  in 
which  procedures  should  be  implemented,  actions  to  be  taken  if  a  proposed  solution  fails, 
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and  how  to  respond  if  necessary  information  is  absent”  (p.  6).  Subsequently,  it  follows 
that  team  performance  will  only  be  effective  when  there  is  shared  strategic  knowledge 
among  team  members  (Stout,  et  al.,  1996).  Another  important  consideration  deals  with 
the  fact  that  declarative  and  procedural  knowledge  remain  static  while  strategic 
knowledge  changes  based  upon  task  parameters  and  team  member  responses  to  task 
events  and  environmental  demands  (Stout,  et  al.,  1994).  In  order  to  avoid  future 
confusion  between  the  cognitive  definition  of  “strategic”  and  the  military  definition,  we 
will  further  refer  to  strategic  knowledge  as  “operational  knowledge”. 

Figure  1,  extracted  from  Stout,  etal.,  (p.  98,  1996)  shows  how  shared  mental 
models  can  be  incorporated  in,  and  transformed  into,  SA.  This  graphic  model  basically 
depicts  three  specific  situations  with  respect  to  shared  mental  models  and  teams.  1)  those 
that  severely  limit  or  exclude  explicit  strategizing,  2)  those  that  allow  for  only  minimal 
and  efficient  explicit  strategizing,  and  3)  those  that  allow  for  unlimited  explicit 
strategizing.  The  term  explicit  strategizing  specifically  refers  to  behaviors  or  processes 
that  team  members  participate  in  for  the  purposes  of  disseminating  information, 
clarifying  actions  and  procedures,  and  planning  for  future  activities  (Stout,  et  al,  1996). 
In  the  case  of  CRM  and  intercockpit  interaction,  this  is  best  communicated  with  verbal 
communication. 

The  flow  of  the  figure  shows  how  teams  draw  from  shared  knowledge  bases  to 
arrive  at  a  certain  level  of  S  A.  Shared  operational  knowledge  allows  for  constant 
validation  and  reassessment  to  take  place.  Feedback  from  previous  events  provides  for 
the  adaptation  and  updating  of  existing  team  models.  Eventually,  team  members  will  be 
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able  to  coordinate  actions  implicitly  as  opposed  to  explicitly.  This  will  have  a  significant 
impact  on  the  ability  of  multiple  aircraft  crews  to  communicate  effectively  and 
efficiently,  and  enhance  SA. 


Figure  1.  A  graphical  representation  of  shared  mental  models 
(Stout,  Cannon-Bowers,  &Salas,  1996) 
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2.3  Situation  Awareness  Defined 


In  the  conduct  of  aviation  operations,  the  ability  of  pilots  to  function  in  three- 
dimensional  space,  process  events  occurring  “beyond  the  cockpit”,  and  react  quickly  and 
appropriately  to  unanticipated  mission  changes  is  imperative.  Within  the  complex  system 
environment,  research  has  suggested  that  a  hypothetical  construct,  which  is  referred  to  as 
“situation  awareness”  (SA),  can  be  used  to  represent  a  pilot’s,  or  any  system  operator’s, 
knowledge  of  the  current  situation  relative  to  the  desired  result  (Bowers,  Braun,  &  Kline, 
1994).  Consequently,  the  lack  of  SA  and/or  the  failure  to  maintain  it  has  been  referenced 
as  a  major  contributing  factor  in  mishaps  involving  complex  systems,  such  as  aircraft, 
and  can  have  catastrophic  repercussions  (Endsley,  1988).  One  analysis  of  military 
aircraft  accidents  that  were  caused  by  human  error  indicated  that  lack  of,  or  problems 
with,  S  A  on  the  part  of  critical  crewmembers  contributed  to  these  mishaps  (Hartel,  Smith, 
&  Prince,  1991). 

Despite  the  fact  that  the  need  to  focus  on  SA  has  been  recognized,  there  has  not 
been  substantial  attention  paid  to  team  SA.  The  preponderance  of  research  efforts  has 
placed  emphasis  on  individual  task  performance  and  SA.  Many  believe  that  this  may  be 
due  to  the  lack  of  definition  clarity  discussed  earlier  (Stout,  et  al. ,  1992)  and  that  it  is 
customary  to  investigate  the  performance  of  individual  team  members  in  a  team  setting  as 
opposed  to  examining  team-level  behavioral  changes  resulting  from  training  (1992). 

Even  less  emphasis  has  been  placed  on  investigating  team  SA  in  an  aircraft-to-aircraft 
environment. 
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In  order  to  discuss  how  SA  contributes  to  overall  aviation  performance,  it  is 
necessary  to  have  a  clear  definition  of  exactly  what  SA  is.  In  many  areas  of  research,  this 
has  been  particularly  difficult  to  obtain.  SA  has  a  notorious  reputation  as  a  fairly  loose 
term,  for  how  does  one  determine  if  an  individual,  or  team  for  that  matter,  is  truly 
“situationally  aware”?  To  arrive  at  an  appropriate  definition,  we  must  define  what  is 
meant  by  a  “situation”.  Pew  (1994)  defines  a  situation  as  “a  set  of  environmental 
conditions  and  system  states  with  which  the  participant  is  interacting  that  can  be 
characterized  uniquely  by  a  set  of  information,  knowledge,  and  response  options”  (p.18. 
Table  1).  Pew  goes  on  to  describe  the  associated  elements  of  a  situation  which 
subsequently  constitute  “awareness”.  The  table  below,  extracted  from  Pew,  provides  a 
synopsis  of  these  elements.  Parenthetical  descriptions  indicate  how  these  elements  would 
apply  to  a  team. 


♦  Current  state  of  the  system  (includes  equipment,  people,  etc.) 

♦  Predicted  state  in  the  future 

♦  Information  and  knowledge  required  to  support  crew  activities 

♦  Activity  phase  (involving  crew/team) 

♦  Prioritized  list  of  goals  (overall  team  objectives) 


Table  1.  Elements  of  Awareness 
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By  combining  these  uses  of  SA,  we  can  apply  them  toward  understanding  a  more 
formal,  and  traditionally  accepted,  definition  put  forth  by  Endsley  (1988,  1995).  He 
defines  S  A  as  “the  perception  of  the  elements  in  the  environment  within  a  volume  of  time 
and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the 
near  future”  (1995,  p.  3 1).  In  expanding  on  this  concept,  Endsley  (1995)  describes  the 
construct  of  SA  in  three  hierarchical  phases,  which  are  shown  in  Table  2: 


♦  Level  ISA-  Perception  of  environmental  elements 

♦  Level  2  SA  -  Current  situation  comprehension 

♦  Level  3  SA  -  Projection  of  future  status 

Table  2.  Levels  of  Situation  Awareness 

Level  1  S  A  is  achieved  by  perceiving,  or  being  provided  specific  data  for  the 
status,  attributes,  and  dynamics  of  relevant  elements  in  the  environment  (Endsley,  1995). 
In  the  case  of  a  military  aviator,  the  perception  of  terrain,  other  aircraft,  enemy  locations 
and  capabilities,  own  aircraft  status,  and  warning  lights  would  be  required.  Additionally, 
the  aviator  must  also  be  able  to  perceive  relevant  characteristics  of  each  of  these  elements 
such  as  color,  quantity  of  enemy  troops  or  equipment,  speed,  etc. 

Achieving  Level  2  SA  “is  based  on  a  synthesis  of  disjointed  Level  1  elements” 
(Endsley,  1995,  p.  31).  This  implies  transcending  the  state  of  basic  awareness  of  present 
elements.  In  Level  2  SA,  knowledge  of  Level  1  elements  are  combined  to  form  patterns 
that  enable  a  decision-maker  to  comprehend  the  significance  of  elements  and  events 
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(1995).  For  example,  a  military  pilot  must  be  able  to  comprehend  how  the  position,  type, 
number,  and  proximity  of  enemy  equipment,  such  as  tanks  or  helicopters,  betrays  or 
indicates  specifics  about  their  intentions.  It  is  important  to  note  that  a  novice  pilot  may 
be  able  to  achieve  the  same  or  a  superior  degree  of  Level  1  SA  as  a  very  experienced 
pilot,  yet  be  incapable  of  integrating  situation  elements  in  order  to  achieve 
comprehension.  Recognition  of  this  fact  is  essential  when  examining  the  results  of 
studies  or  experimentation  involving  SA. 

Level  3  SA  is  concerned  with  “the  ability  to  project  the  future  actions  of  the 
elements  in  the  environment,  at  least  in  the  near  term”  (1995,  p.  32).  This  level  can  only 
be  achieved  through  the  acquisition  of  Level  1  and  2  SA.  Future  projection  allows  a 
decision-maker  to  choose  a  course  of  action  that  will  most  likely  contribute  to  mission 
accomplishment.  If  a  military  pilot  knows  that  a  pair  of  threat  aircraft  is  in  a  particular 
flight  configuration  and  in  a  certain  location,  the  pilot  can  project  in  what  manner  the 
aircraft  are  likely  to  attack  (1995). 

Beyond  recognizing  the  three  levels  of  SA,  we  must  also  consider  other  aspects  of 
SA.  One  of  these  aspects  concerns  the  temporal  nature  of  SA.  It  is  often  built  up  over 
time  and  not  acquired  instantaneously  (1995).  Additionally,  SA  can  be  very  spatial  in 
nature.  According  to  Endsley  (1995),  spatially  determined  aspects  of  a  situation  involve 
the  specification  and  selection  of  which  environmental  aspects  are  important.  He 
describes  this  concept  as  a  series  of  fluid  boundaries  that  are  constantly  changing: 
Situation  awareness  can  be  conceived  of  as  concerning  the  subset  of  the 
environment  which  is  considered  relevant  to  the  task(s)  at  hand.  This 
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boundary  can  be  seen  to  shrink  or  expand  as  various  tasks  present 
themselves....  (p.  32). 

Figure  2,  extracted  from  Endsley  (1995),  depicts  these  boundaries  as  they  may  apply  to  a 
fighter  pilot. 


Figure  2.  Fighter  Pilot  SA 

In  the  context  of  this  discussion,  it  is  also  necessary  to  address  SA  as  it  applies  to 
team  concepts.  Working  together  demands  a  reliance  on  team  member  SA  in  order 
to  effectively  coordinate  actions  and  resources  (Muniz,  Stout,  &  Salas,  1996).  Complex 
environments,  such  as  those  involving  aviation  operations,  inevitably  require  multiple 
individuals,  or  crews,  to  make  decisions  and  perform  tasks.  One  can  conceive  of  overall 
team  SA  as  a  situation  in  which  “each  team  member  will  have  a  specific  set  of  SA 
elements  about  which  they  are  concerned,  as  determined  by  their  responsibilities  within 
the  team”  (1995,  p.  33).  This  implies  a  certain  degree  of  SA  overlap  between  team 
members,  as  depicted  by  Endsley  (1995)  in  Figure  3.  This  overlap  can  also  be  described 
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as  shared  information  that  serves  as  “an  index  of  inter-team  coordination  and  human- 
machine  interface  effectiveness”  (p.34). 


Figure  3.  Team  SA  Overlap. 
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2.4  Communication  and  Team  Situation  Awareness 


There  has  been  substantial  research  in  recent  years  examining  how  various  factors 
affect  SA.  Many  have  suggested  that  attaining  team  SA  is  highly  dependent  upon 
communication  (Palmer,  1990;  Salas,  Prince,  Baker,  &  Shrestha,  1995).  One  study 
conducted  by  Mosier  and  Chidester  (1991)  investigated  how  solicitation  and  transfer 
communication  served  as  indicators  of  SA  during  simulated  in-flight  emergencies  and 
other  activities.  In  this  study,  verbal  communication  was  used  as  an  index  of  team  SA. 
After  analyzing  the  communications  of  23  three-person  crews,  results  showed  that  the 
level  of  team  SA  affected  overall  performance. 

The  relationship  between  communication  and  team  SA  may  have  a  foundation  in 
the  construct  of  shared  mental  models,  a  concept  discussed  earlier.  Salas  et  al.  (1995) 
suggested  that  in  order  for  team  members  to  formulate  relevant  expectations  of  the 
overall  mission/global  task,  each  team  member  must  possess  and  relay  information  that 
contributes  to  this  process.  Developing  these  expectations  can  be  achieved  by  sharing 
pre-existing  knowledge  bases  and  by  communicating  task-relevant  information. 
Additionally,  mental  models  and  strategies  can  be  updated  and  modified  as  team 
members  collect  and  relay  new  information  about  the  task  (Orasanu,  1990). 

Cannon-Bowers  et  al.  (1993)  indicated  that  the  degree  to  which  team  members 
share  mental  models  clearly  affects  task  completion.  Specific  examples  include  a  team’s 
a)  coordination  of  activities,  b)  adaptation  to  dynamic  task  and  team  demands,  c) 
projection  of  team  and  task  needs,  and  d)  prioritization  of  when  to  exchange  pertinent 
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information.  A  recent  study  conducted  by  Muniz,  et  al.  (1996)  examined  the  relationship 
between  communication,  shared  mental  models,  and  SA.  Verbal  responses  across  three 
specified  aspects  of  an  aviation  scenario  (reaction  to  other  aircraft,  air  traffic  control,  and 
weather)  were  used  to  measure  each  team’s  communication  process.  Results  showed  that 
teams  with  no  shared  mental  models  “used  significantly  more  communication  related  to 
general/less  relevant  information  during  high  workload. .  .than  teams  with  shared  mental 
models”  (p.  11).  Additionally,  Muniz  et  al.  (1996)  discovered  that  as  workload  increased, 
both  groups  increased  their  overall  communication  and  modified  their  communication 
processes. 
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2.5  Team  Task  Analysis  and  Performance  Measurement 


As  implied  in  previous  discussions,  a  proper  understanding  of  what  kind  of 
knowledge  is  being  trained  is  key  to  conducting  an  appropriate  task  analysis.  Task 
analysis  approaches,  in  general,  have  been  recognized  as  being  critical  to  the  training 
design  process,  yet,  once  again,  there  has  been  little  emphasis  placed  on  training  at  the 
team  level.  Unfortunately,  this  has  often  resulted  in  the  forced  training  of  individual  tasks 
within  a  team  environment  (Salas  &  Cannon-Bowers,  in  press).  However,  it  is  evident 
today  that  individual  training  is  not  equivalent  to  team  training,  and  there  is  a  definite 
need  for  a  team  task  analysis  tool.  That  is  to  say  “the  team  task  analysis  (or  collective 
task  analysis)  generates  key  information  needed  for  designing  and  delivering  team 
training”  (Seamster,  Rhedding,  &  Kaempf,  1997,  p.  25). 

In  the  case  of  aviation  team  training,  it  is  generally  accepted  that  the  specific  type 
of  knowledge  to  be  trained  is  primarily  operational.  At  this  point,  through  an  analysis  of 
present  training  guidance  and  subject  matter  expert  (SME)  consultation,  it  is  possible  to 
delineate  and  break  down  certain  behaviors  that  can  be  categorized  under  certain  levels  of 
SA.  The  types  of  tasks  chosen  will  depend  upon  the  type  of  aircraft  being  flown,  the  type 
of  mission  to  be  accomplished,  and  the  geography  of  the  operational  scenario  (i.e.,  what 
theater  of  operations  are  we  attempting  to  simulate).  After  determining  these  tasks,  the 
next  critical  phase  in  training  development  and  evaluation  is  the  selection  of  performance 
measures. 
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One  approach  to  addressing  team  performance  measurement,  as  well  as  other 
components  of  teamwork,  that  has  been  extremely  effective  in  recent  studies  is  the 
Targeted  Acceptable  Response  to  Generated  Events  or  Tasks,  or  TARGETS, 
methodology  (Fowlkes,  Lane,  Salas,  Franz,  &  Oser,  1994).  This  particular  approach 
makes  use  of  a  checklist  format  that  collects  structured  observations  of  team  behaviors  as 
they  occur  during  operationally  relevant  scenarios.  The  key  to  this  method  is  the 
identification  of  tasks  that  act  as  indicators  of  specific  team  behaviors  that  are  being 
trained  or  observed  (Oser,  Dwyer,  and  Fowlkes,  1995).  The  tasks  and  behaviors  elicited 
from  these  tasks  are  placed  into  a  TARGETS  checklist.  Items  in  this  list  are  compared  to 
the  behaviors  of  the  subjects  and  are  marked  as  either  present  or  not  present  (1995).  An 
example  of  such  a  checklist  is  illustrated  in  Figure  4  (Fowlkes,  Lane,  Salas,  Franz,  & 
Oser,  1994,  p.  52). 


EVENT 

TARGET 

CRITICAL  AIRCREW 
COORDINATION  BEHAVIOR 

Section  Leader  asks  crew  to 
perform  unsafe  navigation 
procedure  during  preflight  brief 

Pilots  question  unsafe  navigation 
procedure 

State  opinions  on 
decisions/procedures  (under 
Assertiveness) 

Ship’s  helicopter  direction  center 
provides  erroneous  vector  to  first 
waypoint 

Pilots  question  heading 
information 

Ask  question  when  uncertain 
(under  Assertiveness) 

Communications  from  ships 
tower,  aircrewmen,  lead  aircraft 

Pilots  acknowledge 
communications 

Acknowledge  communication 
(under  Communication) 

En  route  navigation 

Pilot-not-flying  provides  heading 
and  distance  for  next  leg  of  flight 
before  waypoint  is  reached 

Provide  information  in  advance 
(under  SA) 

Section  leader  goes  off  course 

Pilot-flying  calls  code  word  for 
“check  navigation” 

Note  deviations  (under  SA) 

Section  leader  calls  for  lead 
change 

Pilot-flying  uses  standard 
terminology  during  lead  change 
Pilot 

Acknowledge  communication 
(under  Communication) 

Figure  4.  Examples  of  events,  TARGETS,  and  associated  critical  crew  behaviors  for 

a  dual  piloted,  military  cargo  helicopter. 
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There  are  positive  aspects  to  using  this  methodology.  One  advantage  lies  in  the 
fact  that  it  is  event-based  (Fowlkes,  Dwyer,  Oser,  &  Salas,  1998).  This  implies  that  in  a 
particular  operational  scenario  stimulus  events  serve  as  cues  for  team  members.  These 
cues  are  designed  to  elicit  certain  behaviors  that  have  been  determined  to  be  important 
through  task  analysis  and  SME  consultation  (1994).  This  practically  removes  the 
necessity  to  have  SMEs  as  observers  for  training  or  experimentation,  since  behaviors 
specified  on  the  TARGETS  checklist  are  either  present  or  they’re  not.  Another  advantage 
of  this  methodology  is  that  it  provides  for  flexibility  in  what  is  tested.  Since  the  number 
of  behaviors  listed  in  a  series  of  TARGETS  checklists  has  no  limitations,  it  is  possible  to 
investigate  many  general  or  a  few  specific  components  of  team  behavior.  This  method 
may  be  useful  in  evaluating  elicited  behaviors  that  suggest  a  level  of  S  A. 

In  addition  to  this  procedure,  there  are  several  other  potential  methods  and 
techniques  available  for  evaluating  S  A  under  experimental  conditions.  The  catch  lies  in 
determining  the  best  and,  hopefully,  least  intrusive  method  to  use  when  examining  team 
training  concepts.  The  ideal  method  would  involve  getting  into  an  operator’s  mind  and 
being  able  to  observe  exactly  what  the  operator  thinks  and  perceives  at  all  times.  Since 
this  is  not  realistic,  we  must  make  use  of  other  available  measuring  tools. 

One  such  category  of  techniques  uses  physiological  measures,  such  as  an 
electroencephalogram  (EEG),  to  determine  whether  information  is  being  registered 
cognitively  (Endsley,  1993).  Though  these  techniques  can  allow  observers  to  determine 
if  elements  in  the  environment  are  being  perceived,  they  do  not  allow  for  the 
“determination  of  how  much  information  remains  in  memory,  if  the  information  is 
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registered  correctly  in  the  mind,  or  what  comprehension  the  subject  has  of  those 
elements”  (p.  81).  For  these  reasons,  current  physiological  techniques  are  not  very  useful 
in  measuring  SA.  However,  it  should  be  noted  that  advancements  in  this  arena  could 
contribute  significantly  to'  our  understanding  and  evaluation  of  SA. 

Another  potential  tool  for  SA  measurement  involves  the  use  of  objective 
performance  measures.  They  can  be  useful  because  they  are  objective  and  generally  non- 
intrusive  (1993).  The  collection  of  data  is  relatively  simple,  and  system  simulation 
components  can  be  programmed  to  record  specific  performance  data  automatically.  They 
exist  in  many  forms  such  as  global  and  external;  however,  these  measures  are  limited 
because  they  either  mask  critical  information  concerning  why  poor  performance  occurs 
(global)  or  can  actually  detract  from  accurately  assessing  SA  because  of  artificial 
alterations  of  the  scenario  (external)  (1993).  These  factors  make  the  use  of  performance 
measures  a  potentially  ineffective  method  for  evaluating  SA. 

Subjective  techniques  can  also  be  used  to  assess  SA.  These  techniques  enable  the 
operator  to  subjectively  rate  their  own  SA  through  a  series  of  rigorous  self  and  system 
evaluations.  The  operator  (self-rating),  the  observer,  or  both  can  complete  these  ratings. 
One  such  self-rating  method  is  the  Situation  Awareness  Rating  Technique  (SART).  This 
method  allows  for  operators  to  rate  a  particular  system  “on  the  amount  of  demand  on 
attentional  resources,  supply  of  attentional  resources,  and  understanding  of  the  situation 
provided.  As  such,  it  considers  operators’  perceived  workload. .  .in  addition  to  their 
perceived  understanding  of  the  situation”  (p.83). 
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A  newer,  more  complex  method  known  as  the  Subjective  Workload  Dominance 
(SWORD)  metric  requires  operators  or  subjects  to  make  “pairwise  comparative  ratings  of 
competing  design  concepts”  (p.  83)  through  the  use  of  a  numerical  scale.  This  continuum 
indicates  the  degree  to  which  a  particular  concept  requires  less  workload  than  another. 

By  incorporating  these  “preferences”  into  an  analytic  hierarchy  process  (AHP), 
evaluators  can  determine  a  numerically  objective  ordering  of  design  and/or  training 
concepts  (1993). 

One  other  method  for  SA  measurement  deals  with  the  use  of  questionnaires.  This 
technique  serves  as  a  more  direct  form  of  SA  measurement  since  it  directs  operators  to 
answer  specific  questions  pertaining  to  SA  indicators  within  the  scenario.  They  may  be 
given  in  a  “post-test”  fashion,  which  is  fairly  non-intrusive,  or  during  a  simulation  as  it  is 
being  run  (freeze  technique)  (1993).  The  Situation  Awareness  Global  Assessment 
Technique  (SAGAT)  is  a  freeze  technique  which  interrupts  the  course  of  a  simulation  in 
order  for  the  subjects  to  complete  survey  questions  relative  to  the  present  state  of  the 
scenario.  Because  this  technique  is  global,  it  includes  queries  that  address  all  three  levels 
of  SA  (1993).  However,  this  technique  is  disadvantaged  in  that  the  freezing  of  the 
simulation  may  be  intrusive  or  performance  altering  in  some  way,  especially  for  certain 
military  and  flight  operations. 
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2.6  Simulation  and  Fidelity 


In  the  development  of  a  training  device  and,  subsequently,  a  training  system,  it  is 
imperative  to  determine  what  level  of  fidelity  is  needed  in  order  to  accomplish  the 
specified  training  goals.  In  addition  to  analyzing  the  tasks  and  objectives  to  be  trained, 
we  need  to  address  the  inclusion  of  an  adequate  level  of  fidelity  in  which  cost 
effectiveness,  cognitive  engineering,  and  team  performance  measurement  come  together 
to  provide  a  maximum  learning  benefit.  As  far  as  fidelity  is  concerned,  more  may  not 
necessarily  be  better.  Salas,  Bowers,  and  Rhodenizer  (1998)  make  this  emphatically 
clear  in  stating  that  the  “best  simulation  in  the  world  does  not  guarantee  training”  and  that 
“in  the  quest  for  a  more  realistic  simulation,  we  may  have  lost  sight  of  the  true  goal  -  a 
more  effective  training  device”  (p.  200). 

As  discussed  earlier,  PC-based  simulation  alone  will  not  a  great  training  device 
make.  Salas,  et  al.  (1998)  point  out  that  simulation  is  a  vital  part  of  aviation  training  that 
is  capable  of  replicating  detailed  terrain  features,  in  flight  emergencies,  weather 
phenomena,  equipment,  and  an  assortment  of  other  conditions.  Yet,  they  go  on  to  discuss 
that  in  order  to  truly  promote  learning  “there  will  need  to  be  a  shift  in  focus  from  the 
design  of  simulation  for  realism  ...  to  the  design  of  human-centered  training  systems  that 
support  the  acquisition  of  complex  skills”  (p.  199). 

In  order  to  determine  the  appropriate  level  of  fidelity  that  should  be  used  in  the 
PC-based  aviation  team  training  realm,  we  should  provide  a  clear  working  definition  of 
what  fidelity  means  and  of  what  it  is  comprised.  The  concept  of  fidelity  has  carried  a 
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rather  fluid  definition  throughout  the  academic  community,  but  as  Hays  and  Singer 
(1989)  relate,  it  “has  most  often  referred  to  the  design  of  simulation  that  is  used  in 
training”  (p.  47).  The  tendency  is  to  use  one  definition  to  cover  all  aspects  and  variables 
associated  with  the  development  of  a  particular  training  device  or  system.  For  the 
purposes  of  this  research,  the  following  definition  of  simulation  fidelity  given  by  Hays 
and  Singer  (1989)  will  be  used  in  the  discussion  of  fidelity  applications: 

Simulation  fidelity  is  the  degree  of  similarity  between  the  training  situation 
and  the  operational  situation  which  is  being  simulated.  It  is  a  two- 
dimensional  measurement  of  this  similarity  in  terms  of:  1)  the  physical 
characteristics,  for  example,  visual,  spatial,  kinesthetic,  etc.;  and  2)  the 
functional  characteristics,  for  example,  the  informational  and  stimulus 
response  options  for  the  training  situation,  (p.  50) 

Of  particular  note  is  the  clear  distinction  between  the  dimensions  of  physical  and 
functional  fidelity.  From  this  definition,  we  can  conclude  that  the  level  of  fidelity  needed 
is  completely  dependent  upon  what  we  intend  to  train.  As  Hays  and  Singer  (1989)  point 
out,  departing  from  a  high  level  of  fidelity  may  not  necessarily  detract  from  the  training 
effectiveness  of  a  simulator,  and  because  “higher  fidelity  is  associated  with  higher  costs, 
it  is  prudent  to  determine  just  how  much  fidelity  is  necessary  in  a  training  situation”  (p. 
52). 

When  considering  aviation  tasks,  the  spectrum  of  simulation  fidelity  can  be  broad. 
If  the  focus  of  the  simulation  is  concerned  with  training  individual  or  procedural  tasks, 
such  as  emergency  procedures,  flight  tasks,  switchology,  aircraft  dynamics,  or  aircraft 
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qualification,  then  the  physical  fidelity  of  the  simulation  environment  would  need  to  be 
very  high.  In  these  instances,  the  emphasis  is  placed  on  enhancing  psychomotor  skills 
and  muscle  memory.  Therefore,  there  exists  a  greater  need  to  replicate  the  actual  aircraft 
environment.  However,  when  considering  team  training,  the  focus  rests  not  so  much  on 
psychomotor  skills  as  team  behavior  and  cognitive  responses  to  stimuli.  In  this  case  we 
are  more  concerned  with  a  high  level  of  functional  fidelity  and  the  simulator’s  ability  to 
provide  the  appropriate  inputs  in  order  for  the  trainees  to  elicit  a  particular  response. 

It  is  safe  to  say  that  neither  extreme  is  truly  sufficient  for  quality  training.  An 
appropriate  balance  of  each  dimension  of  fidelity  is  necessary  in  order  to  train  adequately 
the  given  tasks  at  hand.  Hays  and  Singer  (1989)  confirm  this  point  in  stating  that 
“situations  for  any  task  will  require  some  degree  of  physical  and  functional  fidelity,  but 
the  combination  will  vary  depending  on  the  physical  and  functional  requirements  of  the 
operation”  (p.  310). 

The  additional  complexity  of  helicopter  flight  and  its  environment  adds  to  this 
need  for  balance.  Fixed  wing  aircraft  generally  require  inputs  from  power  controls,  a 
control  yoke  (used  for  turning  the  aircraft  left  or  right  and  for  increasing  and  decreasing 
the  pitch,  or  nose  angle,  of  the  aircraft)  and  pedals  (used  primarily  for  steering  control 
during  ground  operations,  and  for  coordinated  turning  in  air  operations).  Power  settings 
can  be  set  for  a  specified  level  and  the  aircraft  can  basically  be  “trimmed”  to  require  only 
minimal  control  yoke  inputs  during  most  modes  of  flight.  Rotary  wing  aircraft  require 
continual  control  inputs  from  a  cyclic  (left,  right,  pitch  up,  pitch  down),  collective 


30 


(power),  and  pedals.  Any  movement  of  one  of  the  controls  requires  a  corresponding 
movement  in  the  other  two. 

In  order  for  PC-based  simulation  to  be  effective  for  team  training  in  a  helicopter 
environment,  it  must  contain  enough  physical  fidelity  to  adequately  represent  the  mental 
demands  and  flight  responses  to  be  encountered,  yet  still  provide  enough  stimuli  to  elicit 
responses  to  team  training  objectives.  There  has  been  a  significant  amount  of  research 
done  in  recent  years  to  investigate  PC-based  aviation  training  devices  (PCATDs).  Many 
of  these  studies  have  focused  on  the  use  of  PC-based  simulation  to  observe  performance 
in  the  area  of  crew  coordination  and  CRM  (Koonce  &  Bramble,  1998;  Jentsch,  Bowers, 
&  Holmes,  1995;  Jentsch  &  Bowers,  1998;  Stout,  Salas,  &  Merket,  1998;  Bowers,  Salas, 
Prince,  &  Brannick,  1992;  Baker,  Prince,  Shrestha,  Oser,  and  Salas,  1993).  After  using  a 
helicopter  gunship  computer  game  to  observe  crew  communication  patterns,  Bowers,  et 
al.  (1992)  remarked  that  there  could  be  many  advantages  in  using  PC-based  simulation 
for  studying  team  behaviors  and  processes.  Likewise,  Baker,  et  al.  (1993)  discovered 
that  military  pilots  and  crewmembers  favored  the  use  of  PC-based  simulation  for 
reviewing  crew  coordination  tasks  and  crew  resource  management.  In  another  study 
Jentsch  and  Bowers  (1998)  found  that  “most  crews  displayed  performance  behaviors  that 
SMEs  found  characteristic  and  typical  of  aircrews  in  similar  flight  situations”  (p.  247). 
Although  these  studies  focused  primarily  on  intracockpit  team  training,  the  performance 
measures  used  could  be  transferred  to  an  intercockpit  environment  in  order  to  evaluate 
similar  behaviors  in  an  aircraft  to  aircraft  scenario. 
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There  is  also  some  literature  concerning  the  training  effectiveness  of  high  fidelity 
simulation  versus  low  fidelity,  and  consequently  low  cost,  simulation.  Salas,  et  al.  (1998) 
indicated  that  there  is  an  assumption  that  a  high  level  of  scene  detail  and  motion 
contribute  to  “the  realism  of  the  out-of-cockpit  view  that  is  seen  by  the  trainee”  and  that 
this  high  scene  detail  “consequently  enhances  training”  (p.  202).  However,  Salas,  et  al. 
(1998)  go  on  to  reference  other  research  suggesting  that  there  may  be  more  training 
transfer  under  conditions  of  lower  scene  detail  as  opposed  to  high  or  moderate  scene 
detail  (Taylor,  Lintem,  Koonce,  Kaiser,  &  Morrison,  1991),  and  that  ‘complete  physical 
fidelity  is  rarely  required  for  effective  training  and  transfer’  (Flexman  &  Stark,  1987,  p. 
1031).  In  addition,  the  research  of  Stout,  Salas,  and  Merket  (1998)  indicates  that  though 
low-cost,  low-fidelity  simulation  may  not  be  completely  realistic  in  terms  of  physical 
characteristics,  it  can  be  “specifically  designed  to  elicit  desired  KSAs  [knowledge,  skills, 
and  attributes]  thus  maintaining  functional  fidelity”  (p.  311). 

Additionally,  there  is  some  question  as  to  whether  or  not  high  fidelity  simulators 
are  effective  in  evaluating  training.  These  simulators  frequently  “do  not  collect 
performance  measures  that  can  be  readily  used  in  training  evaluation”  (Salas,  et  al.,  1998, 
p.  204).  Low-fidelity  simulation,  however,  may  provide  better  occasions  for  evaluating 
various  levels  of  training  (1998).  Whether  PC-based  simulation  can  be  used  effectively 
for  intercockpit  team  training  remains  to  be  conclusively  proven. 
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2.7  Research  Questions 


In  considering  all  the  complicated  aspects  of  training  system  design,  it  is  clear  that 
developing  a  relevant  and  cost  effective  training  device  is  no  easy  task.  As  the  literature 
review  has  revealed,  more  research  is  needed  in  the  area  of  using  low-cost  simulation  for 
training  U  S.  Army  aviators  in  SA.  Though  there  are  a  multitude  of  issues  to  address  in 
the  development  of  such  a  prototype,  far  too  many  for  the  scope  of  this  research,  the 
information  uncovered  thus  far  does  pose  some  key  questions.  Foremost  among  these 
are:  1)  What  are  the  behaviors  or  skills  that  would  need  to  be  observed  during 
intercockpit  helicopter  operations?,  2)  Can  some  or  all  of  these  behaviors  be  extracted 
from  cases  in  which  the  TARGETS  methodology  has  been  used  to  assess  SA  in 
simulations  involving  intracockpit  activities?,  and  3)  What  conceptual,  low-cost 
framework  can  be  used  to  serve  as  a  reference  point  for  utilizing  the  extracted  TARGETS 
behaviors?  The  remainder  of  this  research  will  address  these  questions. 
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CHAPTER  3 


RESEARCH  METHODOLOGY 
3.1  Research  Concept 


Thus  far  this  research  effort  has  examined  teamwork,  SA,  and  methods  for 
measuring  indicators  of  team  performance  as  they  relate  to  SA.  Now  this  research  effort 
will  address  the  formulation  and  extension  of  behaviors  that  can  be  used  in  a  low-cost 
scenario  to  train  and  assess  US.  Army  aviators  in  the  areas  of  team  performance  and  SA. 
The  methodology  will  consist  of  case-based  “extraction”  from  scenarios  in  other  studies 
that  have  addressed  SA  with  regard  to  aviation  team  training.  Though  these  particular 
cases  may  not  have  specifically  addressed  intercockpit  team  training  and/or  were  not 
conducted  in  a  low-cost  simulation  environment,  these  cases  may  be  useful  in 
determining  if  the  communicated  behaviors  used  in  these  scenarios  can  be  transferred  to 
the  intercockpit  environment,  thus  extending  the  mission  space  of  the  assessment  tool. 
The  case  selection  will  focus  on  those  that  have  utilized  the  TARGETS  methodology  for 
assessing  observable  behaviors  as  they  apply  to  SA.  The  two  cases  that  will  be  examined 
are  Team  Performance  in  Multi-Service  Distributed  Interactive  Simulation  Exercises: 
Initial  Results  (Oser,  R.  L.,  Dwyer,  D.  J.,  &  Fowlkes,  J.  E.,1995)  and  Improving  the 
Measurement  of  Team  Performance:  The  TARGETS  Methodology  (Fowlkes,  J.  E.,  Lane, 
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N.  E.,  Salas,  E.,  Franz,  T.,  &  Oser,  R.  L.,  1994).  Both  of  these  cases  were  addressed  in 
Chapter  2. 

This  research  approach  will  closely  follow  the  methodology  described  in  Robert 
K.  Yin’s  book  Case  Study  Research:  Design  and  Methods;  Second  Edition.  The 
observable  behaviors  extracted  from  the  cases  will  be  modified  as  required  for  Army 
helicopter  operations.  They  will  then  be  placed  into  a  “new”  TARGETS  checklist  and 
reviewed  by  three  Subject  Matter  Experts  (SME)  in  the  rotary  wing  and  U.S.  Army 
aviation  community:  two  Army  instructor  pilots  and  one  retired  Marine  Corps  officer 
with  expertise  in  multi-ship  helicopter  operations.  Should  additional  tasks/behaviors 
relevant  to  Army  multi-ship  rotary  wing  operations  be  required  or  not  required,  they  will 
be  formulated  and  added/deleted  in  accordance  with  Army  doctrine  and  the  foundations 
in  SA  provided  by  the  literature  and  case  reviews. 

In  addition  to  the  development  of  observable  tasks,  the  case  study  effort  will 
consider  the  feasibility  of  performing  an  intercockpit  training  exercise  utilizing  the  newly 
formulated  TARGETS  checklists.  The  scenario  will  be  based  on  an  Aircrew  Training 
Manual  (ATM)  or  other  mission-based  scenario.  It  will  be  analyzed  to  determine  if  it  is 
feasible,  if  the  scenario  exercises  all  of  the  indicated  intercockpit  tasks,  and  if  the 
checklist  supports  the  given  scenario.  In  addition  to  the  scenario  and  TARGETS 
development,  a  low-cost  conceptual  model  formulation  will  also  be  conducted.  This 
model  will  provide  a  framework  in  which  to  use  the  new  TARGETS  assessment  tool.  No 
formal  attempt  to  prove  or  validate  this  conceptual  model  will  be  addressed  during  this 
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research  effort;  however,  suggestions  for  the  validation  and  use  of  such  a  model  will  be 
discussed  in  Chapter  5.  To  collect  data  from  the  indicated  SMEs,  a  questionnaire  will  be 
used  to  address  specific  questions  concerning  the  TARGETS  development. 
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3.2  Case  Study  Approach 


This  case  study  extraction  and  conceptual  model  development  effort  will  consist 
of  three  phases.  These  phases  and  the  critical  activities  that  comprise  each  phase  are 
illustrated  in  Figure  5.  Phase  I,  representing  the  “Problem  Definition  and  Case  Study 
Design”  phase,  involves  defining  the  problem  space,  developing  a  hypothesis  to  address 
the  specified  research  questions,  selecting  specific  cases  to  assist  in  problem-solving,  and 
establishing  the  data  collection  protocol  for  the  case-based  extraction.  Phase  II,  the 
“Preparation,  Collection,  and  Analysis”  phase,  focuses  on  collecting  and  analyzing  data 
from  each  case  study  and  ends  with  a  written  summary  of  the  significant  findings  for 
each  case.  Phase  III,  the  “Analysis  and  Conclusion”  phase,  further  analyzes  the  results  of 
Phase  II  and  then  attempts  to  draw  conclusions  about  the  practical  implications  of 
applying  the  extracted  behaviors  to  an  intercockpit  TARGETS  assessment  tool. 


37 


A8A23ZE& 


Figure  5.  Case  Study  Research  Approach  Critical  Activity  Flow  Diagram 
(Taken  from  Robert  K.  Yin’s  Case  Study  Research:  Design  and  Methods ;  Second 

Edition ) 
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3.3  Phase  I:  Problem  Definition  and  Case  Study  Design 


As  was  suggested  in  Chapters  1  and  2,  there  is  limited  existing  literature  that 
demonstrates  significant  in-depth  research  on  intercockpit  team  training  in  a  low-cost 
environment,  particularly  in  the  area  of  SA.  Focus  on  U.S.  Army  aviation  training  in  this 
capacity  is  equally,  if  not  more,  limited.  Chapter  2  indicates  that  there  is  substantial 
support  for  the  use  of  communication  or  communicated  behavior  as  an  indicator  of  SA. 
Additionally,  Chapter  2  suggests  that  the  TARGETS  methodology  is  a  viable 
measurement  tool  for  evaluating  team  training  concepts,  to  include  S  A.  When 
considering  the  flexible  and  non-intmsive  nature  of  this  methodology,  TARGETS  appears 
to  be  ideal  for  use  in  multiple  U.S.  Army  rotary  and  fixed  wing  platforms.  To  support 
these  assessments,  much  of  the  documentation  cited  referenced  studies  involving 
intracockpit  or  single  crew  coordination  environments.  This  information,  coupled  with 
the  questions  posed  at  the  conclusion  of  Chapter  2,  suggest  the  hypothesis  that 
communicated  behaviors  that  have  been  determined  to  be  appropriate  for  the  intracockpit 
environment  may  be  transferred  to  the  intercockpit  environment  as  an  extension  of  the 
mission  space. 

Based  on  this  hypothesis,  the  next  step  in  this  phase  is  to  select  cases  that  possess 
behaviors/scenarios  that  target  critical  teamwork  skills  that  can  be  accomplished  in  a  low- 
cost,  multi-ship,  environment.  This  will  be  accomplished  through  a  comprehensive 
literature  search.  The  search  will  be  comprised  of  an  examination  of  several  different 
aviation  experimentation  situations  involving  team  S  A  assessment.  Two  cases  selected 
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for  review  are  Team  Performance  in  Multi-Service  Distributed  Interactive  Simulation 
Exercises:  Initial  Results  (Oser,  R.  L.,  Dwyer,  D.  J.,  &  Fowlkes,  J.  E.,1995)  and 
Improving  the  Measurement  of  Team  Performance:  The  TARGETS  Methodology 
(Fowlkes,  J.  E.,  Lane,  N.  E.,  Salas,  E.,  Franz,  T.,  &  Oser,  R.  L.,  1994).  These  cases  were 
selected  due  to  their  specific  use  of  the  TARGETS  methodology  and  their  focus  on 
aviation-related  team  training  and  SA.  After  briefly  analyzing  these  cases,  key  aspects  of 
each  case  will  be  placed  into  a  master  tally  sheet  (See  Figure  4)  that  can  be  referenced  in 
order  to  note  the  specific  differences  between  the  cases. 


Case 

Fidelity 

Simulation  Type 

Aircraft  Type 

TARGETS  Design 

Figure  6.  TARGETS  Analysis  Tally  Sheet 


Following  this  analysis,  specific  tasks  will  be  extracted,  modified  if  necessary, 
and  placed  into  a  new  TARGETS  checklist  based  on  a  generic  scenario  that  would  be 
applicable  and  reasonable  for  Army  multi-ship  helicopter  operations.  For  the  purposes  of 
this  analysis,  the  scenario  will  be  based  upon  a  mission  involving  three  or  more  UH-60 
Blackhawk  helicopters.  This  aircraft  mission  was  selected  due  to  the  generic  nature  of 
the  mission  objectives,  those  in  which  pilots  from  multiple  communities  (attack,  scout, 
etc)  can  reasonably  participate.  It  should  be  noted  that  the  scenario  and  resultant 
TARGETS  checklists  can  be  easily  modified  for  attack  and  reconnaissance  aircraft 
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missions  as  well.  Task  selection  will  be  based  upon  doctrine  suggested  by  Army 
literature,  the  appropriate  aircraft  ATM,  personal  experience,  and  SME  review. 

The  last  step  in  this  phase  is  the  data  collection  protocol.  This  step  contains  the 
procedures  and  general  rules  that  will  be  followed  when  studying  the  two  cases.  In 
general,  the  extraction  of  tasks  from  the  cases  will  be  done  in  accordance  with  U.S.  Army 
aviation  operational  guidelines  outlined  in  doctrinal  publications  and  relevant  aircrew 
training  manuals.  Both  the  TARGETS  assessment  tool  and  the  scenario  will  be 
distributed  to  the  SMEs  for  review  and  modification. 
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3  4  Phase  II:  Preparation.  Collection,  and  Analysis 


Phase  II  will  be  the  actual  case  study  and  task  extraction  phase.  This  phase  will 
prepare,  collect,  and  analyze  data  for  each  case  study.  This  phase  will  be  guided  by  the 
protocol  established  in  section  3.3  and  will  draw  upon  several  different  sources  of 
information  to  ensure  a  thorough  analysis.  This  will  include  the  information  discussed  in 
the  previous  chapter.  Additionally,  as  tasks  are  extracted  from  the  case  studies,  they  will 
be  placed  sequentially  into  the  TARGETS  checklist  that  will  be  subsequently  used  in  an 
intercockpit  training  scenario. 


v 
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3.5  Phase  III:  Analysis  and  Conclusion  Phase 


This  phase  builds  upon  the  analysis  that  ended  Phase  II  and  concludes  with  the 
implications  that  the  case  studies  might  hold  on  the  proposed  hypothesis.  The  findings  in 
this  phase  will  constitute  a  significant  portion  of  Chapter  4  in  this  thesis.  This  phase  will 
augment  the  case-based  task  extraction  discussed  in  the  previous  sections.  This 
augmentation  will  be  done  through  the  use  of  SME  review.  The  newly  developed 
TARGETS  checklist  containing  observable  SA  behaviors  will  be  submitted  to  experts  in 
the  U.S.  Army  rotary  wing  community:  two  instructor  pilots  and  one  former  officer  with 
expertise  in  multi-ship  helicopter  operations.  They  will  provide  feedback  on  these 
checklists  through  the  use  of  a  pre-designed  survey  and  comment  sheet. 

The  review  will  have  the  following  objectives: 

-  Determine  if  the  given  scenario  realistically  represents  characteristics  typical 
of  a  multi-ship  operational  environment. 

Determine  if  the  behaviors  listed  in  the  TARGETS  checklist  accurately 
represent  those  that  would  be  exhibited  in  a  multi-ship  helicopter  environment 
involving  two  or  more  utility  or  attack  helicopters. 

-  Determine  if  there  are  additional  tasks/behaviors  that  should  be  added  to,  or 
deleted  from,  the  scenario. 

Once  the  TARGETS  checklist  has  been  reviewed,  an  analysis  of  the  feedback 
from  the  SMEs  will  determine  whether  or  not,  and/or  to  what  degree,  observable  tasks 
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can  be  extracted  from  other  TARGETS  checklists  used  in  other  simulation  cases.  This 
analysis  will  provide  support  for  or  against  the  proposed  theory  and  research  questions 
posed  in  section  2.7.  The  SME  feedback  will  then  be  used  to  modify  the  draft  TARGETS 
checklist.  The  resultant  product  for  this  thesis  will  be  a  new  TARGETS  checklist  that  can 
be  eventually  used  to  assess  SA  in  U.S.  Army  aviators  participating  in  a  low-cost 
simulation  exercise. 
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3,6  Conceptual  Framework 


Though  the  assessment  tool  can  be  developed  independently  from  the  other 
components  of  this  low-cost  training  system,  it  is  necessary  to  have  a  framework  in 
which  we  expect  to  use  this  tool.  As  indicated  in  section  3. 1,  the  tasks  formulated  for  the 
new  TARGETS  checklist  will  be  extracted  from  cases  that  have  a  varied  base  of  origin: 
from  those  involving  a  single,  high-fidelity  simulator  to  those  involving  multi-service, 
high-cost  distributed  simulation  exercises. 

By  examining  certain  aspects  of  these  and  other  cases  and  the  fidelity  issues 
discussed  in  Chapter  2,  it  is  once  again  possible  to  extract  a  conceptual  framework  that 
can  provide  a  baseline  for  what  type  of  hardware  configuration,  software  requirements, 
and  other  equipment  might  be  sufficient  for  a  low-cost  representation  for  team  training. 
To  achieve  this,  case-based  research  will  be  used  to  formulate  a  conceptual  model  for  a 
low-cost  simulation  environment  that  can  best  serve  the  needs  of  evaluating  team  SA,  yet 
still  be  of  practical  use  for  the  military  in  terms  of  flexibility  and  deployability.  For  the 
purposes  of  this  research,  “low-cost  simulation”  will  be  restricted  to  the  domain  of  PC- 
based  systems,  since  the  PC  would  afford  the  Army  aviation  community  reconfigurability 
and  ease  of  deployability. 

The  cases  chosen  for  analysis  are  Individual  Task  Proficiency  and  Team  Process 
Behavior:  What’s  Important  for  Team  Functioning  (Stout,  Salas,  &  Carson,  1994)  and 
Communication  and  Team  Situational  Awareness  (Bowers,  Braun,  &  Kline,  1994).  A 
logical  question  one  could  pose  is  why  were  these  cases  selected  over  the  myriad  of  other 
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cases  involving  the  use  of  PC-based  simulation?  These  particular  cases  were  selected 
due  to  their  focus  on  subject  areas  relevant  to  this  research  effort.  Both  cases  examine 
one  or  more  aspects  of  team  training,  and  one  case,  Bowers,  et.  al.,  1994,  specifically 
addresses  the  relationship  between  communication  and  SA.  Additionally,  both  cases 
focused  on  aviation-related  skills  and  scenarios.  In  much  the  same  manner  as  the  case- 
based  extraction  methodology  used  to  develop  the  TARGETS  checklists,  key  elements  of 
the  two  cases  given  here  will  be  extracted  and  placed  into  the  conceptual  framework. 

There  is  a  myriad  of  possible  components  that  could  be  considered  in  this  case 
comparison.  However,  it  would  be  unfeasible  to  adequately  address  them  all  in  this 
analysis.  For  the  purposes  of  this  research,  only  the  components  illustrated  in  Figure  7 
will  be  considered  in  each  case.  These  components  have  been  broken  down  into  the 
categories  of  internal  and  external.  The  “Audio”  category  is  overlapping  due  to  the  fact 
that  PC  speakers,  microphones,  and  headsets  may  be  used  as  either  internal  devices  to  the 
system  or  as  an  external  means  of  communication  or  review.  Other  specific  components, 
such  as  graphics  boards  and  sound  cards,  will  not  be  addressed.  A  master  chart  will  be 
used  to  track,  categorize,  and  compare/contrast  each  of  these  elements  (See  Figure  8). 
Added  to  this  checksheet  is  an  additional  column  for  Team  Training/SA.  The  inclusion 
of  this  column  is  to  document  whether  the  case  in  question  does,  indeed,  concern  team 
training  issues,  SA  issues,  or  both.  From  this  analysis,  it  will  be  possible  to  develop  a 
conceptual  model  that  represents  a  low-cost  simulation  environment  in  which  to  apply  the 
TARGETS  assessment  tool. 
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PC  System 
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Figure  7.  Low-Cost  Conceptual  Model  Components 
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Figure  8.  Conceptual  Model  Master  Checksheet 


This  conceptual  model  development  will  follow  the  same  “three-phase”  procedure 
used  for  the  TARGETS  assessment  tool.  The  two  cases  will  be  analyzed,  and  key 
components  will  be  extracted  and  compared  for  possible  application  in  a  low-cost  system. 
It  is  important  to  note  that  the  primary  focus  of  this  research  involves  the  design  of  a 
team  training  assessment  tool  through  the  development  of  a  TARGETS  checklist.  No 
formal  proof  or  validation  of  this  conceptual,  PC-based  framework  will  be  conducted 
beyond  what  has  already  been  done  for  the  conceptual  model  and  limited  user  comments 
on  the  feasibility  and  viability  of  such  a  system. 
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CHAPTER  4 


CASE  RESEARCH  AND  ANALYSIS 

4. 1  TARGETS  Analysis  Case  1 :  Team  Performance  in  Multi-Service  Distributed 
Interactive  Simulation  Exercises Initial  Results 

Since  Desert  Storm,  the  military  has  seen  an  increasing  reliance  upon 
geographically  separated,  multi-service  units  that  must  work  together  to  perform  a  variety 
of  missions  (Oser,  R.  L.,  Dwyer,  D.  I,  &  Fowlkes,  J.  E.,  1995).  This  particular  study 
examined  the  use  of  advancements  in  distributed  interactive  simulation  (DIS) 
technologies  for  use  in  a  Multi-Service  Distributed  Training  Testbed  (MDT2)  exercise  in 
Close  Air  Support  (CAS).  Using  the  TARGETS  methodology,  performance  data  was 
collected  over  the  course  of  the  five-day  exercise.  Ultimately,  recommendations  were 
provided  concerning  the  future  evaluation  of  team  performance  concepts  in  similar  DIS 
environments  (1995). 

The  MDT2  for  this  study  involved  a  system  of  moderate  to  high  fidelity 
simulators  located  at  several  military  training  and  research  facilities  around  the  nation  to 
include  the  Mounted  Warfare  Testbed,  Ft.  Knox;  Armstrong  Laboratory,  Aircrew 
Systems  Training  Division,  Mesa,  A Z;  and  Naval  Command  Control  &  Ocean 
Surveillance  Center  Research  and  Development,  NRaD,  San  Diego,  CA.  The  players 
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consisted  of  a  variety  of  vehicles  and  aircraft  types,  e.g.  helicopters,  F-15s,  and  F-16s. 
Networking  between  the  sites  was  accomplished  through  the  Defense  Simulation  Internet 
(DSInet)  and  the  use  of  T-l  lines.  Additionally,  communications  were  provided  through 
digital  voice  (1995). 

The  scenario  package  included  a  comprehensive  mission  that  was  comprised  of  a 
series  of  separate  flight  segments.  It  began  with  the  mission  brief  and  continued  through 
preflight  and  take-off  procedures  into  the  conduct  of  an  actual  in-flight  mission.  During 
the  mission,  each  site  may  or  may  not  have  been  involved  in  multiple  activities  including 
ingress,  egress,  target  engagement,  air  intercept,  and  attack.  At  the  conclusion  of  the 
mission,  teams  were  debriefed  on  their  performance  in  several  categories,  one  of  which 
being  SA. 

The  desired  behaviors  for  this  exercise  were  represented  in  a  series  of  TARGETS 
checklist  like  the  example  shown  in  Appendix  B.  They  were  designed  very  much  like  a 
standard  TARGETS  checklist,  with  the  exception  of  columns  three  and  four.  Instead  of 
using  a  mark  or  symbol  in  column  three  to  represent  a  “hit”  for  the  indicated  behavior, 
grades  are  placed  in  these  blocks.  This  allows  the  observer  to  not  only  indicate  whether 
or  not  the  desired  behavior  was  observed,  but  also  to  assess  the  degree  of  proficiency  in 
which  the  behavior  was  executed.  The  grading  scheme  used  in  evaluating  the  behaviors 
is  also  shown  in  Appendix  B.  In  this  scale,  the  various  dimensions  being  analyzed  are 
outlined  with  concrete  descriptions  of  what  to  look  for  in  the  behaviors.  Below  this,  a 
numeric  scale  is  given  to  provide  evaluators  with  guidance  for  assigning  grades. 
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Depending  on  the  objectives  of  a  particular  study  or  training  scenario,  evaluators 
may  be  able  to  glean  a  more  precise  assessment  of  a  team’s  SA  through  the  use  of  such  a 
grading  scheme.  Also  of  note  is  the  implementation  of  a  “notes”  column.  This  enables 
the  observer  to  document  key  information,  specify  which  crew  or  crewmember  actually 
elicited  the  behavior,  or  elaborate  on  certain  aspects  of  the  behavior  not  provided  in  the 
“behavior”  column. 
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4.2  TARGETS  Analysis  Case  2:  Improving,  the  Measurement  qf_Team  Performance:  The 


TARGETS  Methodology 

The  goal  of  this  study  was  to  demonstrate  the  use  of  an  event-based  measurement 
technique,  specifically  TARGETS,  in  evaluating  team  performance  for  potential  military 
applications.  The  study  argued  that  stimulus  events  could  be  used  to  serve  as  cues  for 
team  members  in  order  to  elicit  certain  desired  behaviors  that  one  can  place  into 
categories  such  as  leadership,  assertiveness,  decision  making,  and  SA.  The  scenario 
required  six  aircrews  to  fly  a  troop  transport/extraction  mission  in  which  they  would 
encounter  a  variety  of  situations  to  include  change  of  flight  lead  and  lost  communications 
(Fowlkes,  J.  E.,  Lane,  N.  E.,  Salas,  E.,  Franz,  T.,  &  Oser,  R.  L.,  1994).  To  execute  the 
scenario,  the  crews  utilized  a  high-fidelity,  6 -df  motion  base  operational  flight  trainer 
(OFT)  which  replicated  major  aircraft  systems  while  providing  left  and  right  side,  right 
front,  and  chin-bubble  outside  visuals  (1994). 

The  TARGETS  checklists  derived  for  this  particular  study  were  similar  to  those  of 
Case  1.  In  this  instance,  however,  the  checklists  were  designed  for  a  more  objective  and 
quantifiable  approach  to  evaluation.  Each  of  the  events  and  behaviors  was  coded  for  a 
specific  phase  of  the  mission  (e.g.,  preflight)  and  a  specific  aspect  of  performance  (e.g., 
equipment  status  or  information  awareness).  The  number  of  “hits”  for  each  phase  and 
each  category  were  then  recorded  in  a  tally  sheet  that  encapsulates  the  overall  results  of 
the  exercise.  With  this  approach,  evaluators  can  use  the  various  totals  in  each  category  to 
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statistically  quantify  how  the  team  performed.  Examples  of  the  TARGETS  checklist  and 
final  tally  sheet  used  in  this  case  are  shown  in  Appendix  C. 


4.3  TARGETS  Case  Comparison  and  Products 


By  analyzing  the  two  cases  in  sections  4.2  and  4.3,  key  similarities  and 
differences  surface.  These  can  be  noted  by  referencing  the  tally  sheet  below  (see  Figure 
9).  Though  both  studies  were  conducted  in  a  high-fidelity  environment,  each  case  used  a 
different  approach  for  “connecting”  their  teams.  Case  1  used  DSInet  to  connect  the 
various  multi-service  elements,  while  Case  2  used  a  single  OFT.  Both  cases  also  used 
rotary  wing  aircraft  in  their  exercises;  however.  Case  1  was  far  more  complex  and 
involved  the  use  a  variety  of  aircraft  and  vehicles.  Additionally,  each  case  used  a  slightly 
different  approach  to  their  TARGETS  checklist  design.  Case  1  used  a  more  subjective, 
grading-oriented  approach  to  evaluating  behaviors,  while  Case  2  used  an  objective 
“observed  “  or  “not  observed”  technique  for  recording  elicited  behaviors. 


Case 

Fidelity 

Simulation  Type 

Aircraft  Type 

TARGETS  Design 

Team  Performance  in 
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Figure  9.  Final  TARGETS  Analysis  Tally  Sheet 


Through  an  examination  of  Army  literature  and  the  cases  discussed  in  the 
previous  sections  it  is  possible  to  derive  a  viable  scenario  from  which  to  develop  the  new 
events  and  behaviors  for  the  TARGETS  checklist.  The  scenario  constructed  for  this 
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research  is  shown  in  Appendix  D.  It  represents  a  generic  troop  extraction  mission  that  a 
flight  of  Army  utility  helicopters  would  be  expected  to  execute.  It  begins  with  a  mission 
brief  to  flight  crews  by  the  air  mission  commander  (AMC)  and  is  followed  by  a  troop 
extraction  mission  at  some  predetermined  pick-up  zone  (PZ).  The  scenario  ends  with  the 
flight  safely  returning  to  base  after  the  troop  extraction.  During  the  mission,  the  crews 
will  encounter  unexpected  events  to  which  they  must  react  in  a  safe  and  doctrinally 
appropriate  manner.  The  scenario  does  not  specify  exactly  what  these  events  will  be,  but 
does  alert  crews  to  the  fact  that  they  will  most  likely  respond  to  some  in-flight  change  or 
mission  deviation. 

The  events  and  behaviors  that  correspond  to  this  scenario  have  been  placed  into  a 
preliminary  TARGETS  checklist  also  depicted  in  Appendix  D.  The  checklist  format 
represents  a  hybrid  of  the  two  checklists  discussed  in  the  two  previous  sections.  It 
combines  the  subjective  note  and  grading  components  of  the  checklist  from  Case  1,  yet 
allows  for  the  objective  and  straightforward  “present  or  not  present”  approach  used  in 
Case  2.  Evaluators  may  choose  whichever  technique  is  appropriate  for  the  situation.  The 
mission  itself  has  been  broken  down  into  several  segments  that  range  from  the  mission 
brief  to  return  to  base. 

By  comparing  the  tasks  present  in  this  new  checklist  to  those  from  the  two  cases 
discussed  in  this  research,  it  is  evident  that  the  extracted  tasks  are  very  similar  to  those  in 
the  original  studies,  yet  clearly  tailored  to  fit  Army  intercockpit  helicopter  applications. 
One  example  of  this  is  the  use  of  more  multi-ship-oriented  references  such  as  AMC  and 
aircraft  (AC)  as  opposed  to  just  pilot  or  copilot.  Additionally,  for  many  tasks  it  is  not 
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imperative  to  know  exactly  who  demonstrated  a  behavior,  as  in  Case  2,  so  long  as  that 
crew  executed  it  without  being  prompted.  If  the  rater  so  desires,  the  notes  column  can  be 
used  to  specify  who  demonstrated  the  behavior. 

Also  of  note  is  the  clear  distinction  on  the  TARGETS  header  whether  or  not  the 
AMC  is  in  the  rater’s  particular  crew.  The  AMC  bears  all  the  responsibility  for  the 
conduct  of  a  multi-ship  operation,  and  is,  therefore,  required  to  be  aware  of  more 
elements  in  the  environment.  Distinguishing  this  on  the  checklist  allows  the  rater  to 
focus  on  the  AMC  specific  tasks  while  other  crew  raters  can  simply  mark  these  tasks  as 
“not  observed”  or  “not  applicable”  (N/A).  To  assist  raters  with  their  notation  and  provide 
consistency  in  the  evaluation  process,  a  legend  has  been  included  at  the  bottom  of  the 
first  page. 

It  should  also  be  noted  that  not  all  of  the  events  and  behaviors  from  Cases  1  and  2 
could  be  extracted  for  use  in  the  extended  intercockpit  mission  space.  One  key  reason  for 
this  is  the  distinction  between  the  relatively  high-fidelity  simulation  environment  of  the 
cases  and  the  low-cost,  low-fidelity  environment  of  our  training  system.  Without  a 
cockpit  that  replicates  all  the  switchology,  communications  equipment,  and  flight 
controls  of  the  actual  aircraft,  it  is  not  possible  to  do  a  complete  preflight  check  and  taxi. 
Also,  depending  on  the  audio  constraints  of  the  low-cost  system,  communications 
techniques,  such  as  “frequency  hopping”,  will  certainly  be  limited.  The  type  of  hardware 
components  and  audio-visual  support  that  may  be  used  for  this  training  system  will  be 
discussed  in  greater  detail  in  future  sections  concerning  conceptual  model  development. 
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In  addition  to  the  scenario  and  preliminary  TARGETS  checklist,  a  survey  sheet 
has  been  developed  for  use  during  the  SME  review.  This  survey  sheet,  depicted  in 
Appendix  E,  asks  the  SMEs  to  address  the  key  objectives  posed  in  section  3.5. 
Specifically,  it  asks  the  SMEs  to  determine:  1)  if  the  scenario  realistically  represents  a 
mission  conducive  to  Army  multi-ship,  utility  helicopter  operations,  2)  if  the  given  events 
in  the  TARGETS  assessment  tool  follow  an  appropriate  sequence  for  such  a  mission,  and 
3)  if  the  listed  TARGETS  behaviors  are  representative  of  what  multi-ship  crews  would  be 
expected  to  perform.  The  survey  also  provides  comment  space  for  each  question  so  that 
the  SMEs  may  add,  delete,  and/or  make  changes  to  the  current  tasks. 

The  survey  sheet,  scenario,  and  preliminary  TARGETS  checklist  was  forwarded 
to  the  appropriate  SMEs  for  review.  Their  feedback  is  discussed  at  the  end  of  Chapter  4. 
The  TARGETS  products  were  modified  as  necessary  to  reflect  the  changes  suggested  by 
the  SMEs.  After  modifying  these  items  the  resultant  products  consisted  of  a  scenario  and 
accompanying  TARGETS  assessment  tool  to  use  for  training  Army  aviation  is  a  low-cost 
simulation  environment. 
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4.4  SME  Review  of  TARGETS  Checklist 


The  SME  feedback  from  the  TARGETS  assessment  survey  is  shown  in  Appendix 
F.  The  SMEs  unanimously  concurred  that  the  given  scenario  was  representative  of  a 
multi-ship  mission  that  Army  utility  helicopter  aviators  can  expect  to  encounter.  The 
TARGETS  checklist  developed  for  the  hypothesized  low-cost  simulation  framework 
suggests  that  communicated  behaviors  that  have  been  used  in  a  variety  of  other  team 
training  simulation  environments  can  be  extracted  for  use  in  an  intercockpit  environment. 
Though  not  every  task  could  be  transferred  to  the  assessment  tool,  the  SME  review  did 
indicate  that  a  majority  of  the  selected/developed  behaviors  and  tasks  incorporated  into 
the  new  TARGETS  checklist  were  adequate  and  required  only  minor  modifications  to 
reflect  Army  Aviation  specific  considerations. 

Though  they  viewed  the  checklist  as  a  viable  assessment  tool  as  is,  the  SMEs  did 
comment  on  the  seemingly  copilot-oriented  nature  of  the  tasks  and  indicated  that  at  times 
it  was  difficult  to  see  the  multi-ship  application.  This  comment  suggests  that  a  more 
general  checklist  for  one  observer  may  be  better  suited  for  capturing  intercockpit 
behaviors.  The  effectiveness  of  this  approach,  however,  would  be  dependent  on  the 
observer’s  visibility  of  the  crew  stations  and  access  to  the  intercockpit  communications. 
In  addition,  the  SME’s  commented  on  the  potential  need  for  segments  involving 
emergency  procedures  and  enemy  threats. 

It  should  be  noted,  however,  that  this  data  is  representative  of  only  a  small  sample 
of  SMEs  (N=3).  With  such  a  small  sample  size  of  experts,  a  reliable  determination  of 
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statistical  significance  and/or  bias  of  the  survey  is  not  possible.  Though  every  effort  was 
made  to  select  objective  SMEs  with  backgrounds  appropriate  for  assessing  utility 
helicopter  operations,  some  form  of  bias  is  inevitable.  It  is  also  noteworthy  to  point  out 
that  these  products  constitute  only  a  preliminary  research  effort  with  the  objective  of 
developing  an  initial  viable  TARGETS  checklist.  Subsequent,  more  extensive  testing  is 
necessary  in  order  to  legitimize  its  use. 
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4.5  Conceptual  Model  Case  1 :  Individual  Task  Proficiency  and  Team  Process  Behavior 

What ’s  Important  for  Team  Functioning 


This  specific  case  attempted  to  examine  effective  team  performance  as  it  relates  to 
task  proficiency  beyond  individual  team  members  (Stout,  Salas,  &  Carson,  1994). 

During  this  experiment,  male  and  female  undergraduate  students  were  distributed  into 
two-member  crews  consisting  of  a  pilot  and  copilot.  Though  each  crewmember  was 
assigned  specific  tasks  to  accomplish,  coordination  between  team  members  was 
necessary  in  order  to  complete  mission  objectives.  The  crews  were  required  to  “fly”  a 
scenario  with  the  aid  of  a  low-fidelity,  low-cost  flight  simulation  apparatus.  Task 
proficiency  was  measured  in  terms  of  joystick  controllability,  copilot  keyboard  operation, 
and  number  of  targets  killed. 

The  hardware  configuration  selected  for  this  experiment  was  comprised  of  a  low 
cost,  off-the-shelf  (OTS),  PC-based  system,  an  audio  system,  and  a  form  of  video 
recording  system  to  be  used  for  feedback  and  after  action  review  (AAR)  purposes  (1994). 
The  personal  computer  system  included  a  modest  286  IBM  compatible  PC,  four-channel 
video  splitter,  joystick,  keyboard,  and  Micro  Prose’s  Gunship  software.  For  visual 
displays,  three  monitors  were  used.  Two  of  the  monitors  provided  each  crewmember 
with  a  separate  display,  while  the  third  monitor  was  used  to  display  the  team’s 
performance.  The  availability  of  the  channel  splitter  provided  for  simultaneous  viewing 
of  video  images  at  three  separate  locations.  No  haptic  feedback  devices  were  indicated  or 
suggested. 
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The  audio  equipment  used  was  fairly  simple  as  well.  All  components  could  be 
easily  and  inexpensively  purchased  at  a  local  Radio  Shack®  or  audio-visual  supply 
establishment.  In  order  for  the  crewmembers  to  communicate  while  performing  their 
tasks,  they  were  provided  with  Radio  Shack®  lapel  microphones  and  lightweight 
headphone  sets.  With  the  aid  of  a  mixing  board,  the  headphone  sets  and  microphones 
could  be  connected  and  the  output,  subsequently  incorporated  into  the  audio  portion  of 
the  videotape  (to  be  discussed  in  further  detail  below). 

For  monitoring  and  recording  the  performance  of  each  team,  this  experiment 
included  a  portable  video  camera,  a  video  tape  recorder,  connecting  cables  for 
incorporating  the  output  from  the  mixing  board  into  the  recorder,  and  a  basic  black-and- 
white  television  set.  Additionally,  a  tripod  was  used  for  mounting  and  positioning  the 
portable  camera.  This  collection  of  video  components  allowed  for  the  progress  of  the 
team,  which  was  displayed  on  the  third  monitor,  to  be  received  on  the  television. 
Subsequently,  this  picture  could  be  transmitted  from  the  camera  to  the  recorder.  In  order 
to  facilitate  crewmember  communication  and  coordination  for  task  completion,  two 
partitions  were  used  to  separate  the  crewmembers  from  each  other  and  the 
video/observation  station. 


61 


4.6  Conceptual  Model  Case  2:  Cormrmicgtiori  gnd_  Team  Situational  Awareness 


This  particular  study  addressed  the  use  of  communication  patterns  as  a  means  to 
investigate  performance  on  simulated  flight  tasks  engineered  to  demand  a  high  level  of 
SA.  Twenty  instrument  flight  rules  (IFR)  rated  pilots  participated  in  the  study  and  were 
required  to  complete  a  variety  of  tasks  over  the  course  of  a  two-phase  experiment 
(Bowers,  Braun,  &  Kline,  1994).  The  twenty  participants  were  subsequently  broken 
down  into  ten  crews  of  two. 

The  components  selected  for  this  study  were  relatively  simple,  easily  obtainable, 
and  certainly  low-cost.  The  hardware  consisted  of  a  386,  33  MHz  IBM  compatible 
computer  with  keyboard,  optional  joystick,  and  a  single  VGA  monitor.  The  software 
consisted  of  the  low-fidelity  simulation  program  Flight  Simulator  4.0.  For  the  purposes 
of  this  study,  the  software  was  configured  to  represent  a  Cessna  210. 

Crews  were  able  to  view  navigational  aids,  instrument  panel  indicators,  and 
external  visual  scenes  on  the  single  monitor.  Audio  and  recording  equipment  was  limited 
to  the  speakers  accompanying  the  PC  system,  and  a  video  recorder  documenting  the 
actions  of  each  crew.  No  microphones  or  headsets  were  used,  or  were  not  indicated  in 
the  case.  Haptic  feedback  devices  were  also  not  present  or  suggested. 
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4.7  Cross  Case  Comparison 


Through  the  component  analysis  discussed  in  sections  4.5  and  4.6,  each  case’s 
respective  key  elements  have  been  placed  into  the  master  checksheet  alluded  to  in  section 
3.6.  The  resultant  product  of  this  process  is  the  Conceptual  Model  Cross  Comparison 
Checksheet  depicted  in  Appendix  G.  An  examination  and  comparison  of  these  two  cases 
reveals  some  interesting  issues  for  consideration.  Though  each  case  approached  the 
survey  of  team  training  and/or  S  A  concepts  in  much  the  same  manner,  the  complexity  of 
each  case’s  design  is  very  different.  Ultimately  each  case  provided  elements  that  could 
be  applied  to  the  conceptual  model. 

One  item  that  immediately  stands  out  in  this  comparison  is  the  relatively  “slow” 
processing  speed  of  the  chips  used  in  the  PC  configuration.  Though  both  these  studies 
are  more  than  three  years  old,  the  use  of 286  or  386  chips  for  team  training  does  show 
that  the  fastest  and  most  expensive  processor  available  may  not  be  required  for  a  low- 
cost,  low-fidelity  system.  A  moderately  priced  Pentium®  III  based  PC  should  satisfy  the 
needs  of  this  training  system’s  software  and  multi-ship  requirements.  Since  the  new  low- 
cost  system  will  involve  the  participation  of  multiple  aircraft  and  not  just  one  crew,  the 
use  of  one  PC-system  per  aircraft  would  be  required.  Additionally,  the  software  selected 
for  these  systems  should  possess  a  network  capability  for  multiple  player  use.  Microsoft 
Flight  Simulator,  or  a  similar  package,  can  provide  this  type  of  capability. 

Both  cases  suggest  that  some  form  of  video  or  after  action  review  capability  is 
essential  for  monitoring  and  documenting  team  performance.  As  indicated  in  Case  1,  this 
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method  of  recording  team  activities  allows  the  observer  to  review  specific  team  behaviors 
and  communications  after  a  crew  has  completed  the  given  scenario.  This  is  particularly 
helpful  in  facilitating  the  use  of  a  TARGETS  checklist.  By  reviewing  a  video  recording 
of  the  crew’s  performance,  an  observer  may  reevaluate  behaviors  originally  thought  to  be 
present,  but  were  not,  or  provide  credit  for  those  that  may  have  been  present  but  were 
missed. 

The  use  of  visual  displays  in  each  case  is  very  different  as  well.  Though  Case  2 
illustrated  that  one  monitor  can  be  used  for  team  training  applications,  this  design  may  be 
more  suitable  for  intracockpit  team  training  as  opposed  to  intercockpit.  Since  the 
proposed  hypothesis  suggests  an  “extension”  of  the  mission  space  for  multi-ship 
operations,  the  multiple  monitor  setup  demonstrated  in  Case  1  would  be  more  conducive 
for  evaluating  team  concepts  and  communicated  SA  behaviors.  This  is  due  to  the  fact 
that  this  type  of  setup  supports  multiple  player  use,  multiple  visual  scenes,  if  needed,  and 
the  viewing  of  subject  performance  by  an  observer. 

Also  of  note  is  the  use  of  partitions  in  Case  1.  Since  the  literature  and  the  new 
TARGETS  checklist  suggest  that  communicated  behaviors  can  be  fundamental  indicators 
of  team  SA,  the  use  of  partitions  would  facilitate  these  behaviors.  Additionally,  the  use 
of  partitions  could  provide  an  environment  that  is  more  closely  representative  of  what 
elements  are  actually  within  a  crew’s  purview.  Since  an  aircraft  crew  in  an  actual  multi¬ 
ship  environment  cannot  “see”  the  crews  of  other  aircraft,  it  would  be  unrealistic  to 
implement  a  conceptual  model  that  allows  separate  crews  to  view  each  other  and  possibly 
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derive  information  or  increased  SA  through  the  observation  of  facial  expressions  or 
physical  gestures. 

Conspicuously  missing  from  both  cases  was  the  use  of  haptic  devices.  As  the 
Chapter  2  discussion  on  fidelity  suggests,  high  fidelity  may  not  be  as  critical  when  the 
desired  training  is  behavioral  as  opposed  to  psychomotor.  In  both  of  these  cases  the 
focus  centered  on  training  team  performance  concepts,  not  flight  skill  proficiency.  When 
considering  the  implications  of  this  for  a  low-cost  system,  it  follows  that  components 
such  as  haptic  feedback  chairs  or  flight  controls  may  provide  limited  or  even  negligible 
benefit. 

With  these  differences  in  mind,  this  analysis  suggests  that  an  ideal  low-cost,  PC- 
based  system  for  conducting  intercockpit  team  training  would  consist  of  components 
drawn  from  both  cases.  A  depiction  of  this  conceptual  model  is  shown  in  Appendix  G. 
The  components  would  consist  of  the  following: 

-  A  standard  desktop  PC  with  a  minimum  of  a  Pentium®  III  chip. 

-  An  OTS  aviation  software  package  that  can  be  configured  for  a  specific 
aircraft’s  visual  and  instrumentation  displays  and  networked  for  multi-player  use. 

-  Joysticks  and  flight  controls  only  as  needed  or  as  funds  permit. 

-  A  communications  architecture  for  simulation  connectivity. 

-  An  audio/communications  system  consisting  of  PC  speakers,  microphones  and 
headsets. 

-  A  video  recording  system  that  can  monitor  team  flight  performance  and  crew 
behavior. 
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-  Multiple  monitors  and  channel  splitters  (as  required)  for  aircraft  representation 
and  evaluator  observation. 

-  Partitions  to  separate  individual  aircraft  crews  from  each  other  and  the 
evaluators. 

It  is  important  to  note  that  the  connectivity  between  these  individual  crew  stations 
is  based  upon  a  simple  local  area  network  (LAN)  or  modem-based  internet  configuration. 
Due  to  the  fact  that  the  use  of  an  advanced  simulation  communication  protocol,  such  as 
high  level  architecture  (HLA),  would  drive  the  cost  of  the  system  out  of  the  low-cost 
arena,  this  type  of  connectivity  was  not  considered  in  this  research. 
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CHAPTER  5 


CONCLUSIONS 
5.1  Results 


As  indicated  in  the  analysis  in  Chapter  4,  the  SME  review  suggested  that  the 
developed  scenario  and  TARGET  assessment  tool  were  valid  products  that  could  be  used 
to  train  and  evaluate  Army  aviators  in  SA.  Though  their  comments  brought  out  the  need 
for  minor  changes,  such  as  the  addition  of  segments  for  emergency  procedures,  and 
indicated  some  lack  of  clarity  in  discerning  multi-ship  behaviors,  they  did  feel  that  the 
scenario  and  checklist  were  viable  tools. 

This  research  also  proposed  a  low-cost  conceptual  model  in  which  to  use  the  new 
TARGETS  checklist.  An  analysis  of  the  cases  presented  suggested  a  multi-monitor 
system  and  reconfigurable  software  that  could  be  used  with  a  conventional  desktop 
Pentium®  PC.  Of  note  is  the  fact  that  neither  case  made  use  of  haptic  feedback  controls. 
This  point  provides  support  for  the  concept  of  using  lower  fidelity  simulation  for 
behavioral/cognitive  applications.  Though  not  presented  in  either  of  the  cases,  the 
concept  of  multi-monitor  use  for  each  crew  station  may  be  of  considerable  benefit.  This 
configuration  would  allow  a  wider  viewing  area  for  the  pilot(s),  e.g.,  for  acquiring  other 
aircraft  in  the  flight,  and  could  allow  for  a  separate  monitor  for  cockpit  instruments. 
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Although  there  was  no  formal  effort  to  validate  this  model,  it  can  act  as  a  starting  point 
for  further  research.  Approaches  to  this  research  are  discussed  in  the  next  section. 

Despite  the  optimistic  results  obtained  by  this  research,  it  should  be  noted  that  the 
TARGETS  checklist  developed  here  and  the  supporting  low-cost  conceptual  model  are 
“pilot”  products,  and  much  further  in-depth  research  is  needed  to  truly  validate  the 
TARGETS  assessment  tool.  This  would  involve  the  conduct  of  the  given  scenario  in  an 
actual  low-cost  environment  and  a  further  analysis  of  the  task  interreliability.  In  order  to 
assess  whether  or  not  the  tasks  in  the  checklist  are  relevant  for  evaluating  SA,  factors  for 
determining  task  interreliability  would  need  to  be  developed.  Additionally,  an 
application  of  psychometric  theory  may  be  needed  in  order  to  determine  the 
interreliability  among  raters  and  correlation  between  tasks  and  subject  performance 
(Nunnally,  1967). 
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5.2  Lessons  Learned 


The  process  of  conducting  this  research  has  brought  out  some  very  noteworthy 
lessons  learned.  Foremost  among  these  is  the  realization  that  capturing  the  key 
components  and  measurement  of  SA  is  a  tedious  and  difficult  process.  This  is 
exemplified  by  this  checklist’s  difficulty  in  projecting  SA  into  the  future.  A  thorough 
examination  of  S  A  assessment  and  its  application  in  a  low-cost  simulation  environment 
involves  the  integration  of  many  disciplines  such  as  research  psychology,  behavioral 
science,  physiology,  and  computer  science.  To  truly  assess  SA,  each  of  these  areas  must 
be  examined  and  integrated  into  the  research  process  in  order  to  determine  how  each 
factor  affects  the  others.  Even  more  important  is  the  determination  of  a  universal 
definition  of  SA.  In  order  to  develop  a  checklist  that  truly  captures  SA,  one  must  be  able 
to  refer  to  and  rely  upon  a  definition  that  is  accepted  and  well-supported. 

Just  as  the  incorporation  of  multiple  disciplines  would  enhance  experimentation  in 
SA,  the  use  of  multiple  performance  measures  may  also  benefit  the  research  process.  The 
new  TARGETS  checklist  developed  in  this  research  may  have  been  better  supported  by 
the  integration  of  additional  SA  performance  measurement  technique.  One  example  of 
this  would  be  the  use  of  the  SWORD  method  discussed  in  Chapter  2.  Since  the  focus  of 
this  research  did  not  include  the  actual  testing  of  the  checklist  in  a  low-cost  simulation 
environment,  the  relationship  between,  and  the  comparison  of,  the  two  techniques  was 
not  viable.  However,  it  cannot  be  denied  that  the  use  of  multiple  performance 
measurement  techniques  would  lend  greater  credibility  and  reliability  to  SA  assessment. 
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Another  issue  brought  out  in  this  research  was  the  difficulty  in  capturing 
intercockpit  versus  intracockpit  tasks.  Though  the  tasks  in  the  new  TARGETS  checklist 
did  address  communicated  behaviors  between  members  of  different  aircraft  crews,  the 
SME  survey  comments  suggest  that  this  distinction  was  not  always  clear,  and  that  the 
predominance  of  the  tasks  seemed  to  focus  on  intracockpit  activities.  This  indicates  that 
the  line  between  what  is  viewed  as  an  in  or  out-of-cockpit  behavior  is  greatly  dependent 
upon  the  SME’s  opinion  of  the  scenario  and  how  the  scenario’s  tasks  are  sequenced. 
Certain  sections  of  a  mission  may  involve  very  intracockpit-heavy  procedures,  but  still 
require  key  intercockpit  communications  to  demonstrate  multi-ship,  team  SA. 
Determining  a  balance  between  the  two  continues  to  be  a  challenge. 

Checklist  reliability,  as  indicated  in  section  5.1,  may  have  been  improved  as  well. 
Had  time  and  resources  been  available,  more  sound  results  could  have  been  achieved  by 
sending  the  checklist  out  for  SME  review  a  second  time  after  making  changes.  This 
process  would  have  enabled  the  assessment  of  improvement  in  the  checklist 
development.  Additionally,  the  reliability  of  the  SME’s  could  have  been  made  more 
sound  by  inserting  false  or  “bogus”  tasks  to  ensure  that  they  were,  indeed,  reading  the 
detailed  tasks  of  the  checklist  closely. 
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5.3  Areas  for  Future  Research 


As  indicated  earlier  in  this  research,  the  design,  implementation,  and  validation  of 
any  proposed  training  device  are  difficult  tasks  requiring  a  tremendous  scope  of  research 
and  experimentation.  However,  this  research  effort’s  seemingly  limited  emphasis  on  the 
development  of  a  SA  assessment  tool  and  a  preliminary  conceptual  framework  can  act  as 
a  springboard  for  other  relevant  areas  of  research.  One  such  area  involves  the  formal 
validation  of  the  low-cost  conceptual  framework  discussed  in  section  4.7.  A  more 
thorough  examination  of  the  environmental  components  suggested  would  provide  a 
foundation  of  empirical  data  that  will  either  support  or  not  support  the  hypothesized 
conceptual  framework.  This  could  be  accomplished  through  a  two-step  process  involving 
the  testing  of  the  low-cost  platform  itself  and  the  testing  of  the  TARGETS  checklist 
within  the  low-cost  environment.  The  first  step  would  indicate  whether  or  not  the 
derived  components  of  the  conceptual  model  could  support  the  given  multi-ship  scenario 
and  accompanying  events,  behaviors,  and  responses.  Following  this  process,  the 
TARGETS  checklist  would  be  implemented,  using  crews  of  Army  helicopter  pilots,  in 
order  to  determine  the  checklist’s  viability.  The  ultimate  objective  would  be  to 
reexamine  the  checklist  in  an  actual  operational  setting. 

There  are  also  issues  that  surround  the  use  of  the  new  TARGETS  checklist  as  an 
intercockpit  SA  evaluation  tool.  Though  the  tasks  formulated  in  this  research  are  directed 
at  use  in  multi-ship  helicopter  operations,  it  should  be  noted  that  the  intracockpit  aspects 
of  SA  and  other  dimensions  do  not  simply  disappear.  The  SME  survey  comments  and 
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the  discussion  in  section  5.2  concerning  tasks  formulation  difficulty  further  emphasize 
this  fact.  This  notion  suggests  that  the  use  of  multiple  or  composite  TARGETS 
checklists,  those  for  intracockpit  and  intercockpit  assessment,  may  be  valuable  in 
determining  SA  on  several  levels.  Based  upon  the  earlier  discussion  concerning  task 
interreliability,  psychometric  application  would  be  even  more  crucial  in  determining  the 
validity  of  such  a  complex  structure  of  intracockpit  and  intercockpit  behaviors. 

Another  key  area  of  emphasis  for  further  analysis  concerns  evaluating  the  training 
effectiveness  of  such  a  low-cost  training  system  and  assessment  tool.  This  begs  a  query 
as  to  the  intended  use  of  this  low-cost  system  and  assessment  tool.  The  intent  of  this 
research  was  not  to  develop  a  tool  that  could  be  used  to  provide  direct  training  benefit 
from  the  low-cost  environment  to  the  actual  aircraft.  Instead  the  focus  was  on  providing 
a  tool  that  could  be  used  as  a  stepping  stone  to  the  next  level  of  simulation:  the  high- 
fidelity  environment. 

In  order  to  assess  this  effectiveness  accurately,  it  would  be  necessary  to  ascertain 
the  amount  of  training  transfer  that  occurs  in  the  transition  from  the  low-cost,  low  fidelity 
to  a  high-cost,  high  fidelity  simulation  environment,  and,  ultimately,  to  the  actual  aircraft. 
To  research  these  concepts  would  clearly  require  tremendous  scope  and  resources.  Still, 
if  it  can  be  shown  that  an  adequate  level  of  positive  training  transfer  can  be  obtained 
through  the  use  of  a  low-cost  system,  such  as  the  one  proposed  here,  the  implications  for 
monetary  savings,  safety,  pilot  performance,  situation  awareness,  and  ultimately,  mission 
accomplishment,  could  be  invaluable. 
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APPENDIX  A  -  GLOSSARY 
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AAR  -  After  action  review 


AH  -  Attack  Helicopter 

AHP  -  Analytical  hierarchy  process 

AMC  -  Air  Mission  Commander 

APART  -  Annual  proficiency  and  readiness  test 

ARMS  -  Aviation  Reconfigurable  Manned  Simulator 

ARNG  -  Army  National  Guard 

ATM  -  Aircrew  Training  Manual 

AVCATT  -  Aviation  Combined  Arms  Tactical  Trainer 

AVTB  -  Aviation  Test  Bed 

CATT  -  Combined  Arms  Tactical  Trainer 

CAS  -  Close  Air  Support 

CP  -  Copilot 

CRM  -  Crew  Resource  Management 

DA  -  Department  of  the  Army 

df  -  Degrees  of  freedom 

DoD  -  Department  of  Defense 

DIS  -  Distributed  interactive  simulation 

DOTDS  -  Directorate  for  Training,  Doctrine,  and  Simulation 

DSInet  -  Defense  Simulation  Internet 

EEG  -  Electroencephalogram 

FRED  -  Fully  Reconfigurable  Engineering  Device 
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HL  A  -  High  level  architecture 

KSA  -  Knowledge,  skills,  and  attributes 

LAN  -  Local  area  network 

MDT2  -  Multi-Service  Distributed  Training  Testbed 

OFT  -  Operational  flight  trainer 

OTS  -  Off-the-shelf 

PC  -  Personal  computer 

PCATD  -  PC-based  aviation  training  device 

PEO  Avn  -  Program  Executive  Office  for  Aviation 

PIC  -  Pilot-in-command 

PZ  -  Pick-up  zone 

SA  -  Situation  awareness 

SAGAT  -  Situation  Awareness  Global  Awareness  Technique 
SART  -  Situation  Awareness  Rating  Technique 
SIMNET  -  Simulation  Network 
SME  -  Subject  matter  expert 

STRICOM  -  Simulation,  Training,  and  Instrumentation  Command 
SWORD  -  Subjective  Workload  Dominance 

TARGETS  -  Targeted  Acceptable  Response  to  Generated  Events  or  Tasks 
UH  -  Utility  helicopter 

USAAVNC  -  United  States  Army  Aviation  Center 
WP  -  Waypoint 
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APPENDIX  B  -  TARGETS  ASSESSMENT  TOOL  DEVELOPMENT:  CASE  1 

EXAMPLES 
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AAfiES 

.TfMEJF  -  few  phasw  raters  to  tha  briefings  und  planning  that ts  performed  prior  to  #>•  #tait  of  ttw  mi«ion. 

PRIOR  K3  PUSH  refer®  to  the  period  of  time  that  fee  partidportts  ora  operating  in  thoaraa  oast  of  116  degree  s(FEBA) 

INGRESS  -  refers  to  period  of  time  that  the  parttipants  ere  operating  west  of  11$  degress  (FEQA)  but  pror  to  engaging  arty  sir  or 
grouad  targets 

GOAffiAT  AIR  PATROL  FORMAT  OKI  *  refers  to  the  period  of  time  that  participants  are  in  combat  at  patrol  circling  formations  but 

notaygal 

INitmCfePF  ENGAGEMENT  -  refer*  to  Im  period  of  fma  that  participants  are  engaging  an  air  target 
ground  TARGET  attack  -  refer*  to  fee  period  of  ime  mat  parfidparas  are  engaging  a  ground  target 
ESRESS  -  raters  to  the  period  cf  time  that  participants  are  eoresatofl  fee  operating  area 
LtiUHtEF  -  M*er*  to  fee  period  of  urns  that  partopanto  arc  conducting  any  review  of  me  tnttaton. 


COMMUNICATION  -THE  EXCHANGE  OF  INFORMATION  BETWEEN  TWO  OR  MORE  TEAM  MEMBERS 

DESCRIPTORS  -  (T)  iNorm attan  Exchange  «»awtg|y&s  the  dear,  concise,  and  accurate  exchange  of  information  between 
efemenb  of  fee  team,  \Z)  Information  Quifabon  -  involves  the  defection  of  taaccuate  or  incomplete  information  and  taking 
conecthe  action,  (3)  Information  Cadence  -  Invokes  tne  timing,  rhythm,  and  flow  of  information,  and  (4)  Information  Format  - 
IrTVONasfealenriiiTology  and  order  of  the  Information 

SITUATION  AWARENESS  -  THE  EXCHANGE  OF  INFORMATION  USED  TO  DEVELOP  AND  MAINTAIN  AN  ACCURATE 
PERCEPTION  OF  THE  OPERATING  EfMRONMENT 

7 

DESCRIPTORS  -  (1 J  Matialrtaq  or  Overall  Minion  Vow  -  involves  Interactions  involving  mission  goals  and  how  situations! 
factcre  are  etioebng  the  mission  goals,  (2)  Monterino  Mission  Deviations  -  invokes  interaction*  related  to  the  detection  and 
coerrmuwootfon  of  changes  in  the  operational  environ  merits  that  could  affcot  the  mission  plan.  (3}  Mgnftprinq  Mission  Proorees  - 
MVufeoalntcrKtfori*  related  to  the  uuittrtil  fuuatiui  in  id  vtelus  uf  mh»ion  asaeta,  {41  Undcrrta«Ohig  Gurrpnt  Miaaton  Plata  - 
towohie*  interact  iora  that  demnnrfiMfffc  fee  team's  thared  and  com  mo  r,  understanding  nf  the  mission  status,  and  (5)  ^afiSaiPfl 
ftetoo  Mraon  States  -  hnrohres  Interactions  related  to  potential  modfication  of  mMon  plan 

ADAPTABI UTY/F LEKJBI LlTY  -  TiE  EXCHANGE  OF  INFORMATION  RELATED  TO  THE  MODIFICATION  OF  PLANS 

DESCRIPTORS:  <l),Ualntafrwiq  The  PrfrSrfefbd  Plan  -  involves  Interactions  demonstrating  that  the  team  wB  not  modify  Its 
txfctthfl  plan,  (2)  Changing  to  a  Pra-BrtofedAtefnete  Plan  -  invokes  Interactions  demonstrating  that  the  teem  WH  modifying  Its 
currant  plan  to  a  raevwusk  briefed  atomatc  plen.  and  rai  CfienoiTia  la  a  Non-Briefed  Plan  -  Involves  hteractions  tiemonsfratfng 
fee  fears  aril  mcxtfy  is  currant  plan  to  m  plan  feat  ms  not  previously  briefed 

CREW  COORDINATION  •  THE  EXCHANGE  OF  Df FORMATION  RELATED  TO  TEAM  SYNCHRONIZATION 

DCSCRIPTOnfe  |1 )  rVovHfofl  Information  b  Advance  -  Involves  interactions  related  to  team  members  anticipating  team  momban 
needs  for  irlonnatfoo  end  praridiig  fee  tafariratiDii  prior  to  Ms  request,  Proving  BacfoUowlien  Required  -  involve* 

intmctbue  related  to  baa*  members  nraugntetfe  need  for  asststancs  and  providing  such  usteteftoe,  and  (3)  Maintaining 
CtntHflS  -  ffTViHVfiS  intsfaciinns  related  m  mam  rwn#f  imofcft  tatgraaions  armed  upon  durmo  tt«>  Mef  . 


1 

2 

3 

4 

5 

Not  Done/ 
Dangerous 

Below 

Avenge 

Nominal 

Above 

Average 

Extraordinary 
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APPENDIX  C  -  TARGETS  ASSESSMENT  TOOL  DEVELOPMENT:  CASE  2 

EXAMPLES 
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Team  member  verbalized  that  C  l,4*s  call  was  not  for  them  (i.e.,  between  each  other). 
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salient  landmarks  called  out 
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"  MiMion  Objectives  E  ^Demonstrated  Awareness  of  Surrounding  Environment 
OS  *  Orientation  in  Space  P  -  Recognized  Problem 

EX  *  External  Support  A  *  Anticipated  Need  for  Action 

EQ  -  Equipment  Status  T  *  Demonstrated  Knowledge  of  Tasks 

PC  -  Persona!  Capabilities  1  =  Demonstrated  Awareness  of  Information 


APPENDIX  D  -  TARGETS  PRODUCTS 
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DERIVED  CHECKLIST 
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LEGEND:  PIC  -  Pilot-in-command  AMC  -  Air  Mission  Commander  —  -  Behavior  not  observed  PZ  -  Pick-up  Zone 

CP  -  Copilot  /  -  Observed  behavior  /  -  Task  not  applicable  WX  -  Weather 


Crew/Team: _  AMC:  Yes/No  Aircraft  Type: 

Rater:  _  Date: _ 


NOTES 

_ 

HIT / 

BEHAVIOR 

PIC  discussed  fuel  requirements  (e.g.,  “We  have  sufficient 
fuel  for  the  mission") 

-CP  provides  name  and  coordinates  of  each  WP 
-CP  provided  info  before  being  asked  bv  PIC 

PIC  confirmed  proper  entry  of  WP  info 

CP  discussed  strategies  for  calling  out  WP  arrival 

Lead  AC  initiates  communications  check  with  other  AC  at 
the  appropriate  time  prior  to  take-off 

Other  AC  responds  to  communications  check 

-Crew  member  pulled  out  take-off  checklist 
-Crew'  member  asked  other  to  null  out  checklist: 

Specify  who  asked 

-PIC  requests  weather  update  from  AMC 
-PIC  makes  request  before  being  told  by  PC 

AMC  provides  weather  info  to  other  AC 

Lead  AC  requests  take-off  clearance 

Lead  AC  notifies  flight  of  impending  take-off  (e.g.,  “Lead 
pitch  pull  in  3  seconds”) 

-CP  continuously  called  out  take-off  info  (i.c.,  rate  of 
climb,  airspeed,  heading,  etc.) 

-CP  provided  info  before  being  asked  by  PIC 

-CP  logged  departure  time 

-CP  logged  time  before  being  asked  by  PIC 

-CP  starts  clock  for  fuel  check  (15  min) 

-CP  starts  fuel  check  before  being  asked  by  PIC 

EVENT 

WP  procedures 

Communications  check 

Take-off  checklist 

preparation 

Preparation  for  take-off 

Take-off  sequence 

FLIGHT 

SEGMENT 

TAKE-OFF 
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<» 

£ 

NOTES 

* 

Crew/Team:  AMC:  Yes/No  Aircraft 

Rater:  Date:  _ 

HIT 

BEHAVIOR 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  to  WP  1 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  without  being  prompted  by  PIC  or  other  AC 

Other  AC  confirms  heading  and  airspeed  guidance 

-Lead  AC  CP  provides  topographical  description  of  WP  1 
before  flying  over  it 

-Lead  AC  CP  provides  topographical  description  of  WP1 
before  being  prompted  bv  PIC 

-Lead  AC  reported  to  AMC  their  arrival  at  WP  1 
-Lead  AC  replied  to  AMC’s  request  for  position 

-Lead  AC  CP  provides  heading  to  WP  2 

-Lead  AC  CP  provided  heading  without  being  prompted 

bv  PIC 

Other  AC  confirms  heading  and  airspeed  guidance 

AMC  announces  change  of  lead  AC: 

Specify  what  AC  is  now  lead 

-Other  AC  acknowledge  lead  change 

-Other  AC  acknowledge  lead  change  without  being  asked 

bv  AMC 

New  lead  AC  announces  lead  position  (c.g..  “ROCKER 

26  in  front”) 

-Lead  CP  provides  current  heading  info  to  WP  2 
-Lead  CP  provides  current  heading  info  without  being 
asked  by  other  AC 

-Lead  AC  CP  provides  topographical  description  of  WP  2 
before  flying  over  it 

-Lead  AC  CP  provides  topographical  description  of  WP  2 
before  being  prompted  by  PIC 

EVENT 

AC  proceeds  to  WP  1 

AC  proceeds  to  WP  2 

Change  of  light  lead 

' 

FLIGHT 

SEGMENT 

LEG  FROM 
BASE  TO 

WP  1 

LEG  FROM 
WP  1TO 

WP  2 
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-Lead  AC  CP  provides  topographical  description  oi 
before  being  prompted  by  PIC  _ 


AMC:  Yes/No  Aircraft  Type: 

Date: _ 


NOTES 

HIT 

BEHAVIOR 

-Lead  AC  reported  to  AMC  their  arrival  at  WP  2 
-Lead  AC  replied  to  AMCs  request  for  position 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  to  PZ 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  without  being  prompted  bv  PIC  or  other  AC 

Other  AC  confirms  heading  and  airspeed  guidance 

-Lead  AC  CP  provides  topographical  description  of  PZ 
before  landing 

-Lead  AC  CP  provides  topographical  description  of  PZ 
before  being  prompted  by  PIC 

-Lead  AC  reported  to  AMC  their  arrival  at  PZ 
-Lead  AC  replied  to  AMC’s  request  for  position 

AC  reports  pick-up  of  troops  to  the  AMC 

Lead  AC  notifies  flight  of  impending  departure  (e.g., 

“Lead  pitch  pull  in  3  seconds”) 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  to  WP  3 

-Lead  AC  CP  provides  correct  heading  and  airspeed 
guidance  without  being  prompted  by  PIC  or  other  AC 

Other  AC  confirms  heading  and  airspeed  guidance 

AMC  requests  position  report  from  AC _ 

PIC  or  CP  of  AC _ makes  repeated  attempt  to  call  AMC 

PIC  or  CP  of  AC _ announces  to  flight  that 

communications  have  been  lost  with  the  AMC 

EVENT 

AC  proceeds  to  PZ 

Sequence  of  troop 
extraction 

AC  proceeds  to  WP  3 

AMC  requests  position 
report  from  AC _ 

AC _ experiences  lost 

radio  contact  with  the 
AMC 

FLIGHT 

SEGMENT 

LEG  FROM 
WP  2  TO 

PZ 

LEG  FROM 
PZ  TO 

WP  3 
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-Lead  AC  calls  for  WX  and  clearance 

Landing  procedures  -Lead  AC  calls  for  info  without  being  asked  by  AMC 


ROTARY  WING  MULTI-SHIP  SCENARIO  (UTILITY! 


The  following  scenario  is  designed for  use  in  a  PC-based,  low-cost  simulation 
environment.  It  assumes  the  participation  of  a  flight  of  three  or  more  U.S.  Army  utility 
helicopters  and  represents  the  conduct  of  a  generic  troop  extraction  mission. 

MISSION  BRIEF 

All  crews  will  participate  in  a  mission  brief  at  the  base  station  prior  to  “flying”  the 
mission.  The  AMC  will  be  charged  with  heading  the  brief.  Items  to  be  briefed  include, 
but  are  not  limited  to,  crew  and  aircraft  assignments,  weather,  mission  objectives, 
waypoints,  PZ,  and  lost  communication  procedures. 

PREFLIGHT/TAKE-OFF 

Crews  will  report  to  their  designated  aircraft  and  proceed  with  the  appropriate  checklist 
and  communications  procedures.  Following  proper  clearance  from  ATC  (simulated  by 
the  evaluator),  the  flight  will  depart  the  base  station  in  accordance  with  the  appropriate 
checklist  and  take-off  procedures. 

MISSION 

The  flight  will  be  required  to  proceed  to  a  series  of  waypoints  enroute  to  a  predetermined 
PZ  for  troop  pickup.  After  landing  at  the  PZ  and  executing  the  troop  extraction,  the  flight 
will  return  to  the  base  station  via  an  alternate  set  of  waypoints.  During  the  mission, 
crews  may  encounter  unexpected  events  such  as  change  of  flight  lead,  lost 
communications,  or  change  of  mission. 

DEBRIEF 

Following  completion  of  the  mission,  crews  will  be  briefed  on  their  collective 
performance  and  conduct  of  the  mission. 
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APPENDIX  E  -  SME  SURVEY 
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SUBJECT  MATTER  EXPERT  (SME)  SURVEY  SHEET 


SMEName: _  Date: 

Rank/Position: _ 


Please  answer  the  following  questions  as  they  apply  to  the  attached  scenario  and 
TARGETS  checklists.  Annotate  suggested  changes,  additions,  and  deletions  in  the  blanks 
provided  and/or  on  the  actual  documents.  If  additional  comments  are  needed,  place  them 
on  a  sheet  of  paper  and  attach  it  to  this  survey. 

1)  Does  the  given  scenario  accurately  incorporate  essential  tasks  that  might  be  expected 
during  U.S.  Army  multi-ship,  utility  helicopter  operations?  YES/NO 
If  not,  please  annotate. 


2)  Are  the  given  EVENTS  listed  in  the  TARGETS  checklists  implicit  to  the  above 
mission  and  appropriately  sequenced  for  a  multi-ship,  helicopter  mission?  YES/NO 
Are  there  any  missing  or  worthy  of  deletion? 


3)  Are  the  BEHAVIORS  listed  in  the  TARGETS  checklists  tasks  that  crews  would  be 
expected  to  perform  during  a  multi-ship,  helicopter  mission?  YES/NO 


4)  Could  wording  be  improved  anywhere  on  the  TARGETS  checklist  in  order  to  improve 
its  quality?  YES/  NO 

If  so,  please  note  changes  directly  on  the  checklist. 
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APPENDIX  F  -  SME  RESULTS 
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SUBJECT  MATTER  EXPERT  (SME)  SURVEY  SHEET 


Please  answer  the  following  questions  as  they  apply  to  the  attached  scenario  and 
TARGETS  checklists.  Annotate  suggested  changes,  additions,  and  deletions  in  the  blanks 
provided  and/or  on  the  actual  documents.  If  additional  comments  are  needed,  place  them 
on  a  sheet  of  paper  and  attach  it  to  this  survey. 


1)  Does  the  given  scenario  accurately  incorporate  essential  tasks  that  might  be  expected 
during  U.S.  Army  multi-ship,  utility  helicopter  operations? (YE^fNO 
If  not,  please  annotate. 


2)  Are  the  given  EVENTS  listed  in  the  TARGETS  checklists  implicit  to  theabove 
mission  and  appropriately  sequenced  for  a  multi-ship,  helicopter  mission?  tjES/NO 
Are  there  any  missing  or  worthy  of  deletion? 


4)  Could  wording  be  improved  anywhere  on  the  TARGETS  checklist  in  order  to  improve 
its  quality?  YES/  NO 


SUBJECT  MATTER  EXPERT  (SME)  SURVEY  SHEET 


SME  Name:  WkH  t£Q4 _  Date:  B  JoaJ  OD 

Rank/Position:  Me 


Please  answer  the  following  questions  as  they  apply  to  the  attached  scenario  and 
TARGETS  checklists.  Annotate  suggested  changes,  additions,  and  deletions  in  the  blanks 
provided  and/or  on  the  actual  documents.  If  additional  comments  are  needed,  place  them 
on  a  sheet  of  paper  and  attach  it  to  this  survey. 

1)  Does  the  given  scenario  accurately  incorporate  essential  tasks  that  might  be  expected 
during  U.S.  Army  multi-ship,  utility  helicopter  operations?  (YES/NO 
If  not,  please  annotate. 


2)  Are  the  given  EVENTS  listed  in  the  TARGETs  checklists  implicit  to  theabove 
mission  and  appropriately  sequenced  for  a  multi-ship,  helicopter  mission?(YE§)NO 
Are  there  any  missing  or  worthy  of  deletion? 

QmuK  TtHMb  Tiwr  MttbttT  B6  IS  PLAM6  PD2-  AlfcCA-FT _ 

TU-AT  COUlis  fee  EMlt^MTCEeA .  l£-  PAtLUges, 

\jQ66  Of  9t  COUU>  BEl6P-  ACT  ( QMS  TO 

T/MceM,  Dun  es ,  ere.  TlhZ  COULD  R>G  Afcfceo  ro 

P2e PLT  “SfeCTlOJ  OP  CqeCktU^T. 

3)  Are  the  BEHAVIORS  listed  in  the  TARGETS  checklists  task&that  crews  would  be 
expected  to  perform  during  a  multi-ship,  helicopter  mission?  YES)NO 

kJPTEfr  CoQgftl  KiArvO^  CQNOcPTS  IMCOeppgAre^  IMTo  ^OEUAfrO 

Ti\At  'S  feOQ5. _ 


4)  Could  wording  bginiproved  anywhere  on  the  TARGETS  checklist  in  order  to  improve 
its  quality?  YES/(^fCt)  ,  CONOl^O  ,  LOO£iMMfc>  "S&CXaS  TO  fcfc 

If  so,  please  note  changes  directly  on  the  checklist.  I M CO^POtf  A.T && 
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SUBJECT  MATTER  EXPERT  (SME)  SURVEY  SHEET 


SME  Name:  _Mike  Dolezal _  Date:  6/14/00 

Rank/Position:  _LtCol  USMC  (retired) _ 


Please  answer  the  following  questions  as  they  apply  to  the  attached  scenario  and 
TARGETS  checklists.  Annotate  suggested  changes,  additions,  and  deletions  in  the  blanks 
provided  and/or  on  the  actual  documents.  If  additional  comments  are  needed,  place  them 
on  a  sheet  of  paper  and  attach  it  to  this  survey. 

1)  Does  the  given  scenario  accurately  incorporate  essential  tasks  that  might  be  expected 
during  U.S.  Army  multi-ship,  utility  helicopter  operations?  rfflPS/NO 
If  not,  please  annotate. 

Assuming  this  is  a  tactical  mission,  you  may  want  to  address  the  enemy  situation, 
possible  threats,  support  resources  available,  and  other  related  items. 


2)  Are  the  given  EVENTS  listed  in  the  TARGETS  checklists  implicit  to  the  above 
mission  and  appropriately  sequenced  for  a  multi-ship,  helicopter  mission'Jg^^JO 
Are  there  any  missing  or  worthy  of  deletion?  The  EVENTS  are  realistic  for  a  routine 

mission.  You  may  consider  adding  some  items  to  make  an  'other  than  routine'  mission 

and  observe  the  effect  on  situational  awareness. 


3)  Are  the  BEHAVIORS  listed  in  the  TARGETS  checklists  tasksjhat  crews  would  be 

expected  to  perform  during  a  multi-ship,  helicopter  mission'i^^B'/NO 

The  CP  BEHAVIORS  generally  describe  or  separate  the  seasoned,  think-ahead  'sierra 

hotel'  CP  from  the  novice  and  average  CPs. 


4)  Could  wording  b^pproved  anywhere  on  the  TARGETS  checklist  in  order  to  improve 
its  quality?  YES/0O 

If  so,  please  note  changes  directly  on  the  checklist. 
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APPENDIX  G  -  MASTER  CHECKSHEET  AND  DEPICTION  FOR  CONCEPTUAL 

MODEL  CASE  COMPARISON 
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Conceptual  Model  Cross  Comparison  Checksheet 


:  ^  '  i 

mmmmm 

m  ..  .: 

Controls 

|||) 

Process  Behavior 

Team  Training 

286  IBM 
compatible 

MicroProse 

Gunship 

-Keyboard 

-Joystick 

f  asnd  :  1 

Awareness  . 

-Team  Training 
-SA 

386/33  MHz 

IBM  compatible 

Flight  Simulator 
4.0 

-Keyboard 

-Optional 

joystick 

i  i  ;.  . 

:  : 

pa 

(I)  taihvidtial  T^K:-  :;: 
Prbfteieicgr  and.  Tejasi;: 

‘  •  liftscess;  BfelMv|<^|||||| 

3  Total 
-1  per  crew 
member 
-1  for  review  & 
performance 
assessment 

-Portable 

camera 
-Video  tape 
recorder 

-Black-and-white 
TV  set 

-Radio  Shack®  lapel 
microphones 
-Radio  Shack® 
headphone  sets 
-PC  speakers 

Partitions 

':  and  Team  Sitiihijoii f 
Awareness  ''..  • 

1  VGA  monitor 
for  instruments 
&  scenes 

Video  tape 
recorder 

PC  speakers 
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